Cross-orientation summation in texture segregation

Second-order boundaries segment more easily when they are density-defined rather than feature-defined

Previous studies have demonstrated that density is an important perceptual aspect of textural appearance to which the visual system is highly attuned. Furthermore, it is known that density cues not only influence texture segmentation, but can enable segmentation by themselves, in the absence of other cues. A popular computational model of texture segmentation known as the "Filter-Rectify-Filter" (FRF) model predicts that density should be a second-order cue enabling segmentation. For a compound texture boundary defined by superimposing two single-micropattern density boundaries, a version of the FRF model in which different micropattern-specific channels are analyzed separately by different second-stage filters makes the prediction that segmentation thresholds should be identical in two cases: (1) Compound boundaries with an equal number of micropatterns on each side but different relative proportions of each variety (compound feature boundaries) and (2) Compound boundaries with different numbers of micropatterns on each side, but with each side having an identical number of each variety (compound density boundaries). We directly tested this prediction by comparing segmentation thresholds for second-order compound feature and density boundaries, comprised of two superimposed single-micropattern density boundaries comprised of complementary micropattern pairs differing either in orientation or contrast polarity. In both cases, we observed lower segmentation thresholds for compound density boundaries than compound feature boundaries, with identical results when the compound density boundaries were equated for RMS contrast. In a second experiment, we considered how two varieties of micropatterns summate for compound boundary segmentation. In the case where two single micro-pattern density boundaries are superimposed to form a compound density boundary, we find that the two channels combine via probability summation. By contrast, when they are superimposed to form a compound feature boundary, segmentation performance is worse than for either channel alone. From these findings, we conclude that density segmentation may rely on neural mechanisms different from those which underlie feature segmentation, consistent with recent findings suggesting that density comprises a separate psychophysical 'channel'.

Luminance texture boundaries and luminance step boundaries are segmented using different mechanisms

In natural scenes, two adjacent surfaces may differ in mean luminance without any sharp change in luminance at their boundary, but rather due to different relative proportions of light and dark regions within each surface. We refer to such boundaries as luminance texture boundaries (LTBs), and in this study we investigate whether LTBs are segmented using different mechanisms than luminance step boundaries (LSBs). We develop a novel method to generate luminance texture boundaries from natural uniform textures, and using these natural LTB stimuli in a boundary segmentation task, we find that observers are much more sensitive to identical luminance differences which are defined by textures (LTBs) than by uniform luminance steps (LSBs), consistent with the possibility of different mechanisms. In a second and third set of experiments, we characterize observer performance segmenting natural LTBs in the presence of masking LSBs which observers are instructed to ignore. We show that there is very little effect of masking LSBs on LTB segmentation performance. Furthermore, any masking effects we find are far less than those observed in a control experiment where both the masker and target are LSBs, and far less than those predicted by a model assuming identical mechanisms. Finally, we perform a fourth set of boundary segmentation experiments using artificial LTB stimuli comprised of differing proportions of white and black dots on opposite sides of the boundary. We find that these stimuli are also highly robust to masking by supra-threshold LSBs, consistent with our results using natural stimuli, and with our earlier studies using similar stimuli. Taken as a whole, these results suggest that the visual system contains mechanisms well suited to detecting surface boundaries that are robust to interference from luminance differences arising from luminance steps like those formed by cast shadows.

Modeling second-order boundary perception: A machine learning approach

Visual pattern detection and discrimination are essential first steps for scene analysis. Numerous human psychophysical studies have modeled visual pattern detection and discrimination by estimating linear templates for classifying noisy stimuli defined by spatial variations in pixel intensities. However, such methods are poorly suited to understanding sensory processing mechanisms for complex visual stimuli such as second-order boundaries defined by spatial differences in contrast or texture. We introduce a novel machine learning framework for modeling human perception of second-order visual stimuli, using image-computable hierarchical neural network models fit directly to psychophysical trial data. This framework is applied to modeling visual processing of boundaries defined by differences in the contrast of a carrier texture pattern, in two different psychophysical tasks: (1) boundary orientation identification, and (2) fine orientation discrimination. Cross-validation analysis is employed to optimize model hyper-parameters, and demonstrate that these models are able to accurately predict human performance on novel stimulus sets not used for fitting model parameters. We find that, like the ideal observer, human observers take a region-based approach to the orientation identification task, while taking an edge-based approach to the fine orientation discrimination task. How observers integrate contrast modulation across orientation channels is investigated by fitting psychophysical data with two models representing competing hypotheses, revealing a preference for a model which combines multiple orientations at the earliest possible stage. Our results suggest that this machine learning approach has much potential to advance the study of second-order visual processing, and we outline future steps towards generalizing the method to modeling visual segmentation of natural texture boundaries. This study demonstrates how machine learning methodology can be fruitfully applied to psychophysical studies of second-order visual processing.

Many naturally occurring visual boundaries are defined by spatial differences in features other than luminance, for example by differences in texture or contrast. Quantitative models of such “second-order” boundary perception cannot be estimated using the standard regression techniques (known as “classification images”) commonly applied to “first-order”, luminance-defined stimuli. Here we present a novel machine learning approach to modeling second-order boundary perception using hierarchical neural networks. In contrast to previous quantitative studies of second-order boundary perception, we directly estimate network model parameters using psychophysical trial data. We demonstrate that our method can reveal different spatial summation strategies that human observers utilize for different kinds of second-order boundary perception tasks, and can be used to compare competing hypotheses of how contrast modulation is integrated across orientation channels. We outline extensions of the methodology to other kinds of second-order boundaries, including those in natural images.

Binocular functional architecture for detection of contrast-modulated gratings

Combination of signals from the two eyes is the gateway to stereo vision. To gain insight into binocular signal processing, we studied binocular summation for luminance-modulated gratings (L or LM) and contrast-modulated gratings (CM). We measured 2AFC detection thresholds for a signal grating (0.75c/deg, 216ms) shown to one eye, both eyes, or both eyes out-of-phase. For LM and CM, the carrier noise was in both eyes, even when the signal was monocular. Mean binocular thresholds for luminance gratings (L) were 5.4dB better than monocular thresholds - close to perfect linear summation (6dB). For LM and CM the binocular advantage was again 5-6dB, even when the carrier noise was uncorrelated, anti-correlated, or at orthogonal orientations in the two eyes. Binocular combination for CM probably arises from summation of envelope responses, and not from summation of these conflicting carrier patterns. Antiphase signals produced no binocular advantage, but thresholds were about 1-3dB higher than monocular ones. This is not consistent with simple linear summation, which should give complete cancellation and unmeasurably high thresholds. We propose a three-channel model in which noisy monocular responses to the envelope are binocularly combined in a contrast-weighted sum, but also remain separately available to perception via a max operator. Vision selects the largest of the three responses. With in-phase gratings the binocular channel dominates, but antiphase gratings cancel in the binocular channel and the monocular channels mediate detection. The small antiphase disadvantage might be explained by a subtle influence of background responses on binocular and monocular detection.

Pooling of first-order inputs in second-order vision

The processing of texture patterns has been characterized by a model that first filters the image to isolate one texture component, then applies a rectifying nonlinearity that converts texture variation into intensity variation, and finally processes the resulting pattern with mechanisms similar to those used in processing luminance-defined images (spatial-frequency- and orientation-tuned filters). This model, known as FRF for filter rectify filter, has the appeal of explaining sensitivity to second-order patterns in terms of mechanisms known to exist for processing first-order patterns. This model implies an unexpected interaction between the first and second stages of filtering; if the first-stage filter consists of narrowband mechanisms tuned to detect the carrier texture, then sensitivity to high-frequency texture modulations should be much lower than is observed in humans. We propose that the human visual system must pool over first-order channels tuned to a wide range of spatial frequencies and orientations to achieve texture demodulation, and provide psychophysical evidence for pooling in a cross-carrier adaptation experiment and in an experiment that measures modulation contrast sensitivity at very low first-order contrast.

Texture sparseness, but not local phase structure, impairs second-order segmentation

Texture boundary segmentation is typically thought to reflect a comparison of differences in Fourier energy (i.e. low-order texture statistics) on either side of a boundary. However in a previous study (Arsenault, Yoonessi, & Baker, 2011) we showed that the distribution of energy within a natural texture (i.e. its higher-order statistical structure) also influences segmentation of contrast boundaries. Here we examine the influence of specific higher-order texture statistics on segmentation of contrast- and orientation-defined boundaries. Using naturalistic synthetic textures to manipulate the sparseness, global phase structure, and local phase alignments of carrier textures, we measure segmentation thresholds based on forced-choice judgments of boundary orientation. We find a similar pattern of results for both contrast and orientation boundaries: (1) randomizing all structure by globally phase scrambling the texture reduces segmentation thresholds substantially, (2) decreasing sparseness also reduces thresholds, and (3) removing local phase alignments has little or no effect on segmentation thresholds. We show that a two-stage filter model with an intermediate compressive nonlinearity and expansive output nonlinearity can account for these data using synthetic textures. Furthermore, the model parameter fits obtained using synthetic textures also predict the segmentation thresholds presented in Arsenault, Yoonessi, and Baker (2011) for natural and phase-scrambled natural texture carriers.

Inconsistent channel bandwidth estimates suggest winner-take-all nonlinearity in second-order vision

The processing of texture patterns has been characterized by a model that postulates a first-stage linear filter to highlight a component texture, a pointwise rectification stage to convert contrast for the highlighted texture into mean response strength, followed by a second-stage linear filter to detect the texture-defined pattern. We estimated the spatial-frequency bandwidth of the second-stage filter mediating orientation discrimination of orientation-modulated second-order gratings by measuring threshold elevation in the presence of filtered noise added to the modulation signal. This experiment yielded no evidence for frequency tuning. A second experiment, in which subjects had to detect similar second-order gratings while judging their modulation frequency, produced bandwidth estimates of 1-1.5 octaves, similar to estimated bandwidths of first-order channels. We propose that an additional dominant-response-selection nonlinearity can account for these apparently contradictory results.

Paradoxical psychometric functions ("swan functions") are explained by dilution masking in four stimulus dimensions

The visual system dissects the retinal image into millions of local analyses along numerous visual dimensions. However, our perceptions of the world are not fragmentary, so further processes must be involved in stitching it all back together. Simply summing up the responses would not work because this would convey an increase in image contrast with an increase in the number of mechanisms stimulated. Here, we consider a generic model of signal combination and counter-suppression designed to address this problem. The model is derived and tested for simple stimulus pairings (e.g. A + B), but is readily extended over multiple analysers. The model can account for nonlinear contrast transduction, dilution masking, and signal combination at threshold and above. It also predicts nonmonotonic psychometric functions where sensitivity to signal A in the presence of pedestal B first declines with increasing signal strength (paradoxically dropping below 50% correct in two-interval forced choice), but then rises back up again, producing a contour that follows the wings and neck of a swan. We looked for and found these "swan" functions in four different stimulus dimensions (ocularity, space, orientation, and time), providing some support for our proposal.

A common rule for integration and suppression of luminance contrast across eyes, space, time, and pattern

Visual perception begins by dissecting the retinal image into millions of small patches for local analyses by local receptive fields. However, image structures extend well beyond these receptive fields and so further processes must be involved in sewing the image fragments back together to derive representations of higher order (more global) structures. To investigate the integration process, we also need to understand the opposite process of suppression. To investigate both processes together, we measured triplets of dipper functions for targets and pedestals involving interdigitated stimulus pairs (A, B). Previous work has shown that summation and suppression operate over the full contrast range for the domains of ocularity and space. Here, we extend that work to include orientation and time domains. Temporal stimuli were 15-Hz counter-phase sine-wave gratings, where A and B were the positive and negative phases of the oscillation, respectively. For orientation, we used orthogonally oriented contrast patches (A, B) whose sum was an isotropic difference of Gaussians. Results from all four domains could be understood within a common framework in which summation operates separately within the numerator and denominator of a contrast gain control equation. This simple arrangement of summation and counter-suppression achieves integration of various stimulus attributes without distorting the underlying contrast code.

Spatial arrangement in texture discrimination and texture segregation

We investigated the role of spatial arrangement of texture elements in three psychophysical experiments on texture discrimination and texture segregation. In our stimuli, oriented Gabor elements formed an iso-oriented and a randomly oriented texture region. We manipulated (1) the orientation similarity in the iso-oriented region by adding orientation jitter to the orientation of each Gabor; (2) the spatial arrangement of the Gabors: quasi-random or regular; and (3) the shape of the edge between the two texture regions: straight or curved. In Experiment 1, participants discriminated an iso-oriented stimulus from a stimulus with only randomly oriented elements. Experiment 2 required texture segregation to judge the shape of the texture edge. Experiment 3 replicated Experiment 2 with Gabors of a smaller spatial extent in a denser arrangement. We found comparable performance levels with regular and quasi-random Gabor positions in the discrimination task but not in the segregation tasks. We conclude that spatial arrangement plays a role in a texture segregation task requiring shape discrimination of the texture edge but not in a texture discrimination task in which it is sufficient to discriminate an iso-oriented region from a completely random region.

Orientation gradient detection exhibits variable coupling between first- and second-stage filtering mechanisms

We investigated sensitivity to orientation modulation using visual stimuli with bandpass filtered noise carriers. We characterized the relationship between the spatial parameters of the modulator and the carrier using a 2-AFC detection task. The relationship between these two parameters is potentially informative of the underlying coupling between first- and second-stage filtering mechanisms, which, in turn, may bear on the interrelationship between striate and extrastriate cortical processing. Our previous experiments on analogous motion stimuli found an optimum sensitivity when the ratio of the carrier and modulator spatial frequency parameters (r) was approximately ten. The current results do not exhibit an optimum sensitivity at a given value of the ratio r. Previous experiments involving second-order modulation sensitivity show an inconsistent range of estimates of optimum sensitivity at values of r between 5 and 50. Our results, using a complementary approach, confirm these discrepancies, demonstrating that the coupling between carrier and modulator frequency parameters depends on a number of stimulus-specific factors, such as contrast sensitivity, stimulus eccentricity, and absolute values of the carrier and modulator spatial frequency parameters. We show that these observations are true for a stimulus limited in eccentricity and that this orientation-modulated stimulus does not exhibit scale invariance. Such processing can not be modeled by a generic filter-rectify-filter model.

Beyond multiple pattern analyzers modeled as linear filters (as classical V1 simple cells): useful additions of the last 25 years

This review briefly discusses processes that have been suggested in the last 25 years as important to the intermediate stages of visual processing of patterns. Five categories of processes are presented: (1) Higher-order processes including FRF structures; (2) Divisive contrast nonlinearities including contrast normalization; (3) Subtractive contrast nonlinearities including contrast comparison; (4) Non-classical receptive fields (surround suppression, cross-orientation inhibition); (5) Contour integration.

Texture modulation detection by probability summation among orientation-selective and isotropic mechanisms

Substantial evidence has accumulated for the notion that modulations of second-order properties in the visual scene are processed by mechanisms which detect contrast variations within narrow orientation/spatial frequency channels. It has also been suggested that mechanisms exist which detect contrast modulations across all orientations. Many naturally occurring texture variations (e.g., modulations in orientation and/or spatial frequency) involve simultaneous contrast modulations in multiple channels. Contrasting conclusions have been drawn regarding the manner in which the information carried in multiple channels is combined. In a series of two experiments it is shown that simultaneous contrast modulations in two narrow orientation bands are detected by three mechanisms, two of which detect contrast modulations within the modulated bands only, the third of which integrates contrast across orientations in order to detect modulations of overall contrast. The three mechanisms combine their efforts by probability summation.

Area summation in human vision at and above detection threshold

The initial image-processing stages of visual cortex are well suited to a local (patchwise) analysis of the viewed scene. But the world's structures extend over space as textures and surfaces, suggesting the need for spatial integration. Most models of contrast vision fall shy of this process because (i) the weak area summation at detection threshold is attributed to probability summation (PS) and (ii) there is little or no advantage of area well above threshold. Both of these views are challenged here. First, it is shown that results at threshold are consistent with linear summation of contrast following retinal inhomogeneity, spatial filtering, nonlinear contrast transduction and multiple sources of additive Gaussian noise. We suggest that the suprathreshold loss of the area advantage in previous studies is due to a concomitant increase in suppression from the pedestal. To overcome this confound, a novel stimulus class is designed where: (i) the observer operates on a constant retinal area, (ii) the target area is controlled within this summation field, and (iii) the pedestal is fixed in size. Using this arrangement, substantial summation is found along the entire masking function, including the region of facilitation. Our analysis shows that PS and uncertainty cannot account for the results, and that suprathreshold summation of contrast extends over at least seven target cycles of grating.

Differential roles of contrast polarity reveal two streams of second-order visual processing

Humans can easily segregate texture regions based on differences in contrast, orientation, and contrast polarity. It has been suggested that these abilities can be inclusively modeled by 2nd-order visual mechanisms that detect changes in the half-wave rectified outputs of orientation-selective filters. Using a subthreshold-summation paradigm, however, we show that modulations of contrast polarity are detected by mechanisms that pool signals of different orientations while modulations of orientation are detected by mechanisms that pool signals of different contrast polarities. The results support the existence of two streams of 2nd-order processing, one that receives the full-wave rectified inputs from oriented filters, the other separate half-wave rectified outputs from on-center and off-center filters pooled across all orientations. The two-stream model is shown to predict the perceptual effects of changes to the skewness statistics of natural-image textures, and to solve a contradiction among previous data concerning the detection of contrast modulation.