Spatiochromatic statistics of natural scenes: first- and second-order information and their correlational structure

Color and luminance processing in V1 complex cells and artificial neural networks

Object recognition by natural and artificial visual systems benefits from the identification of object boundaries. A useful cue for the detection of object boundaries is the superposition of luminance and color edges. To gain insight into the suitability of this cue for object recognition, we examined convolutional neural network models that had been trained to recognize objects in natural images. We focused specifically on units in the second convolutional layer whose activations are invariant to the spatial phase of a sinusoidal grating. Some of these units were tuned for a nonlinear combination of color and luminance, which is broadly consistent with a role in object boundary detection. Others were tuned for luminance alone, but very few were tuned for color alone. A literature review reveals that V1 complex cells have a similar distribution of tuning. We speculate that this pattern of sensitivity provides an efficient basis for object recognition, perhaps by mitigating the effects of lighting on luminance contrast polarity. The absence of a contrast polarity-invariant representation of chromaticity alone suggests that it is redundant with other representations.

Emergence of crowding: The role of contrast and orientation salience

Crowding causes difficulties in judging attributes of an object surrounded by other objects. We investigated crowding for stimuli that isolated either S-cone or luminance mechanisms or combined them. By targeting different retinogeniculate mechanisms with contrast-matched stimuli, we aim to determine the earliest site at which crowding emerges. Discrimination was measured in an orientation judgment task where Gabor targets were presented parafoveally among flankers. In the first experiment, we assessed flanked and unflanked orientation discrimination thresholds for pure S-cone and achromatic stimuli and their combinations. In the second experiment, to capture individual differences, we measured unflanked detection and orientation sensitivity, along with performance under flanker interference for stimuli containing luminance only or combined with S-cone contrast. We confirmed that orientation sensitivity was lower for unflanked S-cone stimuli. When flanked, the pattern of results for S-cone stimuli was the same as for achromatic stimuli with comparable (i.e. low) contrast levels. We also found that flanker interference exhibited a genuine signature of crowding only when orientation discrimination threshold was reliably surpassed. Crowding, therefore, emerges at a stage that operates on signals representing task-relevant featural (here, orientation) information. Because luminance and S-cone mechanisms have very different spatial tuning properties, it is most parsimonious to conclude that crowding takes place at a neural processing stage after they have been combined.

Segmenting surface boundaries using luminance cues

Segmenting scenes into distinct surfaces is a basic visual perception task, and luminance differences between adjacent surfaces often provide an important segmentation cue. However, mean luminance differences between two surfaces may exist without any sharp change in albedo at their boundary, but rather from differences in the proportion of small light and dark areas within each surface, e.g. texture elements, which we refer to as a luminance texture boundary. Here we investigate the performance of human observers segmenting luminance texture boundaries. We demonstrate that a simple model involving a single stage of filtering cannot explain observer performance, unless it incorporates contrast normalization. Performing additional experiments in which observers segment luminance texture boundaries while ignoring super-imposed luminance step boundaries, we demonstrate that the one-stage model, even with contrast normalization, cannot explain performance. We then present a Filter–Rectify–Filter model positing two cascaded stages of filtering, which fits our data well, and explains observers' ability to segment luminance texture boundary stimuli in the presence of interfering luminance step boundaries. We propose that such computations may be useful for boundary segmentation in natural scenes, where shadows often give rise to luminance step edges which do not correspond to surface boundaries.

Color improves edge classification in human vision

Despite the complexity of the visual world, humans rarely confuse variations in illumination, for example shadows, from variations in material properties, such as paint or stain. This ability to distinguish illumination from material edges is crucial for determining the spatial layout of objects and surfaces in natural scenes. In this study, we explore the role that color (chromatic) cues play in edge classification. We conducted a psychophysical experiment that required subjects to classify edges into illumination and material, in patches taken from images of natural scenes that either contained or did not contain color information. The edge images were of various sizes and were pre-classified into illumination and material, based on inspection of the edge in the context of the whole image from which the edge was extracted. Edge classification performance was found to be superior for the color compared to grayscale images, in keeping with color acting as a cue for edge classification. We defined machine observers sensitive to simple image properties and found that they too classified the edges better with color information, although they failed to capture the effect of image size observed in the psychophysical experiment. Our findings are consistent with previous work suggesting that color information facilitates the identification of material properties, transparency, shadows and the perception of shape-from-shading.

A systematic comparison between visual cues for boundary detection

The detection of object boundaries is a critical first step for many visual processing tasks. Multiple cues (we consider luminance, color, motion and binocular disparity) available in the early visual system may signal object boundaries but little is known about their relative diagnosticity and how to optimally combine them for boundary detection. This study thus aims at understanding how early visual processes inform boundary detection in natural scenes. We collected color binocular video sequences of natural scenes to construct a video database. Each scene was annotated with two full sets of ground-truth contours (one set limited to object boundaries and another set which included all edges). We implemented an integrated computational model of early vision that spans all considered cues, and then assessed their diagnosticity by training machine learning classifiers on individual channels. Color and luminance were found to be most diagnostic while stereo and motion were least. Combining all cues yielded a significant improvement in accuracy beyond that of any cue in isolation. Furthermore, the accuracy of individual cues was found to be a poor predictor of their unique contribution for the combination. This result suggested a complex interaction between cues, which we further quantified using regularization techniques. Our systematic assessment of the accuracy of early vision models for boundary detection together with the resulting annotated video dataset should provide a useful benchmark towards the development of higher-level models of visual processing.

Encoding and estimation of first- and second-order binocular disparity in natural images

•

First- and second-order responses to natural binocular images are correlated.

•

Second-order mechanisms can improve the accuracy of disparity estimation.

•

Second-order mechanisms can extend the depth range of binocular stereopsis.

The first stage of processing of binocular information in the visual cortex is performed by mechanisms that are bandpass-tuned for spatial frequency and orientation. Psychophysical and physiological evidence have also demonstrated the existence of second-order mechanisms in binocular processing, which can encode disparities that are not directly accessible to first-order mechanisms. We compared the responses of first- and second-order binocular filters to natural images. We found that the responses of the second-order mechanisms are to some extent correlated with the responses of the first-order mechanisms, and that they can contribute to increasing both the accuracy, and depth range, of binocular stereopsis.

Cue Combination of Conflicting Color and Luminance Edges

Abrupt changes in the color or luminance of a visual image potentially indicate object boundaries. Here, we consider how these cues to the visual "edge" location are combined when they conflict. We measured the extent to which localization of a compound edge can be predicted from a simple maximum likelihood estimation model using the reliability of chromatic (L-M) and luminance signals alone. Maximum likelihood estimation accurately predicted the pattern of results across a range of contrasts. Predictions consistently overestimated the relative influence of the luminance cue; although L-M is often considered a poor cue for localization, it was used more than expected. This need not indicate that the visual system is suboptimal but that its priors about which cue is more useful are not flat. This may be because, although strong changes in chromaticity typically represent object boundaries, changes in luminance can be caused by either a boundary or a shadow.

Detection of chromatic and luminance distortions in natural scenes

A number of studies have measured visual thresholds for detecting spatial distortions applied to images of natural scenes. In one study, Bex [J. Vis.10(2), 1 (2010)10.1167/10.2.231534-7362] measured sensitivity to sinusoidal spatial modulations of image scale. Here, we measure sensitivity to sinusoidal scale distortions applied to the chromatic, luminance, or both layers of natural scene images. We first established that sensitivity does not depend on whether the undistorted comparison image was of the same or of a different scene. Next, we found that, when the luminance but not chromatic layer was distorted, performance was the same regardless of whether the chromatic layer was present, absent, or phase-scrambled; in other words, the chromatic layer, in whatever form, did not affect sensitivity to the luminance layer distortion. However, when the chromatic layer was distorted, sensitivity was higher when the luminance layer was intact compared to when absent or phase-scrambled. These detection threshold results complement the appearance of periodic distortions of the image scale: when the luminance layer is distorted visibly, the scene appears distorted, but when the chromatic layer is distorted visibly, there is little apparent scene distortion. We conclude that (a) observers have a built-in sense of how a normal image of a natural scene should appear, and (b) the detection of distortion in, as well as the apparent distortion of, natural scene images is mediated predominantly by the luminance layer and not chromatic layer.

Texture variations suppress suprathreshold brightness and colour variations

Discriminating material changes from illumination changes is a key function of early vision. Luminance cues are ambiguous in this regard, but can be disambiguated by co-incident changes in colour and texture. Thus, colour and texture are likely to be given greater prominence than luminance for object segmentation, and better segmentation should in turn produce stronger grouping. We sought to measure the relative strengths of combined luminance, colour and texture contrast using a suprathreshhold, psychophysical grouping task. Stimuli comprised diagonal grids of circular patches bordered by a thin black line and contained combinations of luminance decrements with either violet, red, or texture increments. There were two tasks. In the Separate task the different cues were presented separately in a two-interval design, and participants indicated which interval contained the stronger orientation structure. In the Combined task the cues were combined to produce competing orientation structure in a single image. Participants had to indicate which orientation, and therefore which cue was dominant. Thus we established the relative grouping strength of each cue pair presented separately, and compared this to their relative grouping strength when combined. In this way we observed suprathreshold interactions between cues and were able to assess cue dominance at ecologically relevant signal levels. Participants required significantly more luminance and colour compared to texture contrast in the Combined compared to Separate conditions (contrast ratios differed by about 0.1 log units), showing that suprathreshold texture dominates colour and luminance when the different cues are presented in combination.

Form-cue invariant second-order neuronal responses to contrast modulation in primate area V2

A fundamental task of the visual system is to extract figure-ground boundaries between images of objects, which in natural scenes are often defined not only by luminance differences but also by "second-order" contrast or texture differences. Responses to contrast modulation (CM) and other second-order stimuli have been extensively studied in human psychophysics, but the neuronal substrates of second-order responses in nonhuman primates remain poorly understood. In this study, we have recorded single neurons in area V2 of macaque monkeys, using both CM patterns as well as conventional luminance modulation (LM) gratings. CM stimuli were constructed from stationary sine wave grating carrier patterns, which were modulated by drifting envelope gratings of a lower spatial frequency. We found approximately one-third of visually responsive V2 neurons responded to CM stimuli with a pronounced selectivity to carrier spatial frequencies, and often orientations, that were clearly outside the neurons' passbands for LM gratings. These neurons were "form-cue invariant" in that their tuning to CM envelope spatial frequency and orientation was very similar to that for LM gratings. Neurons were tuned to carrier spatial frequencies that were typically 2-4 octaves higher than their optimal envelope spatial frequencies, similar to results from human psychophysics. These results are distinct from CM responses arising from surround suppression, but could be understood in terms of a filter-rectify-filter model. Such neurons could provide a functionally useful and explicit representation of segmentation boundaries as well as a plausible neural substrate for human perception of second-order boundaries.

Possible functions of contextual modulations and receptive field nonlinearities: pop-out and texture segmentation

When analyzing a visual image, the brain has to achieve several goals quickly. One crucial goal is to rapidly detect parts of the visual scene that might be behaviorally relevant, while another one is to segment the image into objects, to enable an internal representation of the world. Both of these processes can be driven by local variations in any of several image attributes such as luminance, color, and texture. Here, focusing on texture defined by local orientation, we propose that the two processes are mediated by separate mechanisms that function in parallel. More specifically, differences in orientation can cause an object to "pop out" and attract visual attention, if its orientation differs from that of the surrounding objects. Differences in orientation can also signal a boundary between objects and therefore provide useful information for image segmentation. We propose that contextual response modulations in primary visual cortex (V1) are responsible for orientation pop-out, while a different kind of receptive field nonlinearity in secondary visual cortex (V2) is responsible for orientation-based texture segmentation. We review a recent experiment that led us to put forward this hypothesis along with other research literature relevant to this notion.

Scotopic hue percepts in natural scenes

Traditional trichromatic theories of color vision conclude that color perception is not possible under scotopic illumination in which only one type of photoreceptor, rods, is active. The current study demonstrates the existence of scotopic color perception and indicates that perceived hue is influenced by spatial context and top-down processes of color perception. Experiment 1 required observers to report the perceived hue in various natural scene images under purely rod-mediated vision. The results showed that when the test patch had low variation in the luminance distribution and was a decrement in luminance compared to the surrounding area, reddish or orangish percepts were more likely to be reported compared to all other percepts. In contrast, when the test patch had a high variation and was an increment in luminance, the probability of perceiving blue, green, or yellow hues increased. In addition, when observers had a strong, but singular, daylight hue association for the test patch, color percepts were reported more often and hues appeared more saturated compared to patches with no daylight hue association. This suggests that experience in daylight conditions modulates the bottom-up processing for rod-mediated color perception. In Experiment 2, observers reported changes in hue percepts for a test ring surrounded by inducing rings that varied in spatial context. In sum, the results challenge the classic view that rod vision is achromatic and suggest that scotopic hue perception is mediated by cortical mechanisms.

Differential effects of exogenous and endogenous attention on second-order texture contrast sensitivity

The visual system can use a rich variety of contours to segment visual scenes into distinct perceptually coherent regions. However, successfully segmenting an image is a computationally expensive process. Previously we have shown that exogenous attention--the more automatic, stimulus-driven component of spatial attention--helps extract contours by enhancing contrast sensitivity for second-order, texture-defined patterns at the attended location, while reducing sensitivity at unattended locations, relative to a neutral condition. Interestingly, the effects of exogenous attention depended on the second-order spatial frequency of the stimulus. At parafoveal locations, attention enhanced second-order contrast sensitivity to relatively high, but not to low second-order spatial frequencies. In the present study we investigated whether endogenous attention-the more voluntary, conceptually-driven component of spatial attention--affects second-order contrast sensitivity, and if so, whether its effects are similar to those of exogenous attention. To that end, we compared the effects of exogenous and endogenous attention on the sensitivity to second-order, orientation-defined, texture patterns of either high or low second-order spatial frequencies. The results show that, like exogenous attention, endogenous attention enhances second-order contrast sensitivity at the attended location and reduces it at unattended locations. However, whereas the effects of exogenous attention are a function of the second-order spatial frequency content, endogenous attention affected second-order contrast sensitivity independent of the second-order spatial frequency content. This finding supports the notion that both exogenous and endogenous attention can affect second-order contrast sensitivity, but that endogenous attention is more flexible, benefitting performance under different conditions.

The developing visual system is not optimally sensitive to the spatial statistics of natural images

The adult visual system is optimally tuned to process the spatial properties of natural scenes, which is demonstrated by sensitivity to changes in the 1/f(α) amplitude spectrum. It is also well documented that different aspects of spatial vision, including those likely responsible for the perception of natural scenes (e.g., spatial frequency discrimination), do not become mature until late childhood. This led us to hypothesise that the developing visual system is not optimally tuned to process the spatial properties of real-world scenes. The present study investigated how sensitivity to the statistical properties of natural images changes during development. Thresholds for discriminating a change in the slope of the amplitude spectrum of a natural scene with a reference α of 0.7, 1.0, or 1.3 where measured in children aged 6, 8, and 10 years (n=16 per age) and in adults (mean age=23). Consistent with previous studies, adults were least sensitive for the shallowest α (i.e., 0.7) and most sensitive for the steepest α (i.e., 1.3). Six- and 8-year-olds had significantly higher discrimination thresholds compared to the 10-year-olds and adults for α's of 1.0 and 1.3, and 10-year-olds did not differ significantly from adults for any of the α's tested. These data suggest that sensitivity to detecting a change in the spatial characteristics of natural scenes during childhood may not be optimally tuned to the statistics of natural images until about 10 years of age. Rather, is seems that perception of natural images could be limited by the known immaturities in spatial vision (Ellemberg, Lepore, & Turgeon, 2010). The question remains as to whether the adult's exquisite sensitivity to the spatial properties of the natural world is experience driven or whether it is part of our genetic programming that only fully expresses itself in late childhood.

Contrast edge colors under different natural illuminations

Essential to sensory processing in the human visual system is natural illumination, which can vary considerably not only across space but also along the day depending on the atmospheric conditions and the sun's position in the sky. In this work, edges derived from the three postreceptoral Luminance, Red-Green, and Blue-Yellow signals were computed from hyperspectral images of natural scenes rendered with daylights of Correlated Color Temperatures (CCTs) from 2735 to 25,889 K; for low CCT, the same analysis was performed using Planckian illuminants up to 800 K. It was found that average luminance and chromatic edge contrasts were maximal for low correlated color temperatures and almost constants above 10,000 K. The magnitude of these contrast changes was, however, only about 2% across the tested daylights. Results suggest that the postreceptoral opponent and nonopponent color vision mechanisms produce almost constant responses for color edge detection under natural illumination.

The perceptual balance of color

The cone contrasts carrying different dimensions of color vision vary greatly in magnitude, yet the perceived contrast of color and luminance in the world appears similar. We examined how this perceptual balance is adjusted by adaptation to the contrast in images. Observers set the level of L vs. M and S vs. LM contrast in 1/f noise images to match the perceived strength of a fixed level of luminance contrast. The perceptual balance of color in the images was roughly consistent with the range of contrast characteristic of natural images. Relative perceived contrast could be strongly biased by brief prior exposure to images with lower or higher levels of chromatic contrast. Similar adaptation effects were found for luminance contrast in images of natural scenes. For both, observers reliably chose the contrast balance that appeared correct, and these choices were rapidly recalibrated by adaptation. This recalibration of the norm for contrast could reflect both changes in sensitivity and shifts in criterion. Our results are consistent with the possibility that color mechanisms adjust the range of their responses to match the range of signals in the environment, and that contrast adaptation plays an important role in these adjustments.

Exogenous attention enhances 2nd-order contrast sensitivity

Natural scenes contain a rich variety of contours that the visual system extracts to segregate the retinal image into perceptually coherent regions. Covert spatial attention helps extract contours by enhancing contrast sensitivity for 1st-order, luminance-defined patterns at attended locations, while reducing sensitivity at unattended locations, relative to neutral attention allocation. However, humans are also sensitive to 2nd-order patterns such as spatial variations of texture, which are predominant in natural scenes and cannot be detected by linear mechanisms. We assess whether and how exogenous attention--the involuntary and transient capture of spatial attention--affects the contrast sensitivity of channels sensitive to 2nd-order, texture-defined patterns. Using 2nd-order, texture-defined stimuli, we demonstrate that exogenous attention increases 2nd-order contrast sensitivity at the attended location, while decreasing it at unattended locations, relative to a neutral condition. By manipulating both 1st- and 2nd-order spatial frequency, we find that the effects of attention depend both on 2nd-order spatial frequency of the stimulus and the observer's 2nd-order spatial resolution at the target location. At parafoveal locations, attention enhances 2nd-order contrast sensitivity to high, but not to low 2nd-order spatial frequencies; at peripheral locations attention also enhances sensitivity to low 2nd-order spatial frequencies. Control experiments rule out the possibility that these effects might be due to an increase in contrast sensitivity at the 1st-order stage of visual processing. Thus, exogenous attention affects 2nd-order contrast sensitivity at both attended and unattended locations.

Beyond multiple pattern analyzers modeled as linear filters (as classical V1 simple cells): useful additions of the last 25 years

This review briefly discusses processes that have been suggested in the last 25 years as important to the intermediate stages of visual processing of patterns. Five categories of processes are presented: (1) Higher-order processes including FRF structures; (2) Divisive contrast nonlinearities including contrast normalization; (3) Subtractive contrast nonlinearities including contrast comparison; (4) Non-classical receptive fields (surround suppression, cross-orientation inhibition); (5) Contour integration.

Statistics of natural scenes and cortical color processing

We investigate the spatial correlations of orientation and color information in natural images. We find that the correlation of orientation information falls off rapidly with increasing distance, while color information is more highly correlated over longer distances. We show that orientation and color information are statistically independent in natural images and that the spatial correlation of jointly encoded orientation and color information decays faster than that of color alone. Our findings suggest that: (a) orientation and color information should be processed in separate channels and (b) the organization of cortical color and orientation selectivity at low spatial frequencies is a reflection of the cortical adaptation to the statistical structure of the visual world. These findings are in agreement with biological observations, as form and color are thought to be represented by different classes of neurons in the primary visual cortex, and the receptive fields of color-selective neurons are larger than those of orientation-selective neurons. The agreement between our findings and biological observations supports the ecological theory of perception.

Region grouping in natural foliage scenes: image statistics and human performance

This study investigated the mechanisms of grouping and segregation in natural scenes of close-up foliage, an important class of scenes for human and non-human primates. Close-up foliage images were collected with a digital camera calibrated to match the responses of human L, M, and S cones at each pixel. The images were used to construct a database of hand-segmented leaves and branches that correctly localizes the image region subtended by each object. We considered a task where a visual system is presented with two image patches and is asked to assign a category label (either same or different) depending on whether the patches appear to lie on the same surface or different surfaces. We estimated several approximately ideal classifiers for the task, each of which used a unique set of image properties. Of the image properties considered, we found that ideal classifiers rely primarily on the difference in average intensity and color between patches, and secondarily on the differences in the contrasts between patches. In psychophysical experiments, human performance mirrored the trends predicted by the ideal classifiers. In an initial phase without corrective feedback, human accuracy was slightly below ideal. After practice with feedback, human accuracy was approximately ideal.

Independence of color and luminance edges in natural scenes

Form vision is traditionally regarded as processing primarily achromatic information. Previous investigations into the statistics of color and luminance in natural scenes have claimed that luminance and chromatic edges are not independent of each other and that any chromatic edge most likely occurs together with a luminance edge of similar strength. Here we computed the joint statistics of luminance and chromatic edges in over 700 calibrated color images from natural scenes. We found that isoluminant edges exist in natural scenes and were not rarer than pure luminance edges. Most edges combined luminance and chromatic information but to varying degrees such that luminance and chromatic edges were statistically independent of each other. Independence increased along successive stages of visual processing from cones via postreceptoral color-opponent channels to edges. The results show that chromatic edge contrast is an independent source of information that can be linearly combined with other cues for the proper segmentation of objects in natural and artificial vision systems. Color vision may have evolved in response to the natural scene statistics to gain access to this independent information.

Is color patchy?

In many natural scenes, shadows and shading, which are primarily luminance-defined features, proliferate. Hence one might expect that the chromatic variations of natural scenes, which more faithfully represent the layout of object surfaces, will contain relatively fewer and larger uniform regions than the luminance variations, i.e., will be more "patchy." This idea was tested using images of natural scenes that were decomposed into chromatic and luminance layers modeled as the responses of the red-green, blue-yellow, and luminance channels of the human visual system. Patchiness was defined as the portion of pixels falling within a +/- threshold in the bandpass-filtered image, averaged across multiple filter scales. The red-green layers were found to be the most patchy, followed by the blue-yellow layers, with the luminance layers the least patchy. The correlation between image-layer patchiness and the slope of the Fourier amplitude spectrum was small and negative for all layers, the maximum value being for red-green (-0.48). We conclude that the chromatic layers of natural scenes contain more uniform areas than the luminance layers and that this is unpredicted by the slope of the Fourier amplitude spectrum.

Seasonal variations in the color statistics of natural images

We examined how the distribution of colors in natural images varies as the seasons change. Images of natural outdoor scenes were acquired at locations in the Western Ghats, India, during monsoon and winter seasons and in the Sierra Nevada, USA, from spring to fall. The images were recorded with an RGB digital camera calibrated to yield estimates of the L, M, and S cone excitations and chromatic and luminance contrasts at each pixel. These were compared across time and location and were analyzed separately for regions of earth and sky. Seasonal climate changes alter both the average color in scenes and how the colors are distributed around the average. Arid periods are marked by a mean shift toward the +L pole of the L vs. M chromatic axis and a rotation in the color distributions away from the S vs. LM chromatic axis and toward an axis of bluish-yellowish variation, both primarily due to changes in vegetation. The form of the change was similar at the two locations suggesting that the color statistics of natural images undergo a characteristic pattern of temporal variation. We consider the implications of these changes for models of both visual sensitivity and color appearance.

Ecologically valid combinations of first- and second-order surface markings facilitate texture discrimination

Natural scenes contain localized variations in both first-order (luminance) and second-order (contrast and texture) information. There is much evidence that first- and second-order stimuli are detected by distinct mechanisms in the mammalian visual system. However, in natural scenes the two kinds of information tend to be spatially correlated. Do correlated and uncorrelated combinations of first- and second-order stimuli differentially affect perception? To address this question we employed orientation-modulated textures in which observers were required to discriminate the spatial frequency of the texture modulation. The textures consisted of micropatterns defined as either local variations in luminance (first-order) or luminance contrast (second-order). Performance was robust with textures composed of only first-order micropatterns, but impossible with only second-order micropatterns. However, when the second-order micropatterns were combined with the first-order micropatterns, they enhanced performance when the two were spatially correlated, but impaired performance when the two were spatially uncorrelated. We conclude that local second-order information may enhance texture modulation discrimination provided it is combined with first-order information in an ecologically valid manner.