Properties of second-order spatial frequency channels

Spatial frequency channels depend on stimulus bandwidth in normal and amblyopic vision: an exploratory factor analysis

The Contrast Sensitivity Function (CSF) is the measure of an observer's contrast sensitivity as a function of spatial frequency. It is a sensitive measure to assess visual function in fundamental and clinical settings. Human contrast sensitivity is subserved by different spatial frequency channels. Also, it is known that amblyopes have deficits in contrast sensitivity, particularly at high spatial frequencies. Therefore, the aim of this study was to assess whether the contrast sensitivity function is subtended by the same spatial frequency channels in control and amblyopic populations. To determine these spatial frequency channels, we performed an exploratory factor analysis on five datasets of contrasts sensitivity functions of amblyopic and control participants measured using either gratings or noise patches, taken from our previous studies. In the range of 0.25-10 c/d, we identified two spatial frequency channels. When the CSF was measured with noise patches, the spatial frequency channels presented very similar tuning in the amblyopic eye and the fellow eye and were also similar to what was observed in controls. The only major difference was that the weight attributed to the high frequency channel was reduced by approximately 50% in the amblyopic eye. However, when the CSF was measured using gratings, the spatial frequency channels of the amblyopic eye were tuned toward lower spatial frequencies. These findings suggest that there is no mechanistic deficit for contrast sensitivity in amblyopia and that amblyopic vision may just be subjected to excessive internal noise and attenuation at higher spatial frequencies, thereby supporting the use of therapeutic strategies that involve rebalancing contrast.

Second-order boundaries segment more easily when they are density-defined rather than feature-defined

Previous studies have demonstrated that density is an important perceptual aspect of textural appearance to which the visual system is highly attuned. Furthermore, it is known that density cues not only influence texture segmentation, but can enable segmentation by themselves, in the absence of other cues. A popular computational model of texture segmentation known as the "Filter-Rectify-Filter" (FRF) model predicts that density should be a second-order cue enabling segmentation. For a compound texture boundary defined by superimposing two single-micropattern density boundaries, a version of the FRF model in which different micropattern-specific channels are analyzed separately by different second-stage filters makes the prediction that segmentation thresholds should be identical in two cases: (1) Compound boundaries with an equal number of micropatterns on each side but different relative proportions of each variety (compound feature boundaries) and (2) Compound boundaries with different numbers of micropatterns on each side, but with each side having an identical number of each variety (compound density boundaries). We directly tested this prediction by comparing segmentation thresholds for second-order compound feature and density boundaries, comprised of two superimposed single-micropattern density boundaries comprised of complementary micropattern pairs differing either in orientation or contrast polarity. In both cases, we observed lower segmentation thresholds for compound density boundaries than compound feature boundaries, with identical results when the compound density boundaries were equated for RMS contrast. In a second experiment, we considered how two varieties of micropatterns summate for compound boundary segmentation. In the case where two single micro-pattern density boundaries are superimposed to form a compound density boundary, we find that the two channels combine via probability summation. By contrast, when they are superimposed to form a compound feature boundary, segmentation performance is worse than for either channel alone. From these findings, we conclude that density segmentation may rely on neural mechanisms different from those which underlie feature segmentation, consistent with recent findings suggesting that density comprises a separate psychophysical 'channel'.

Second-order texture gratings produce overestimation of height in depictions of rectangles and steps

The horizontal-vertical illusion (HVI) has been proposed as a method to increase the perceived height of steps, increase toe clearance and prevent falls. High contrast vertical stripes are placed on the step riser abutting a horizontal edge-highlighter creating 'T' junctions which are thought to promote the illusion. Various configurations of the HVI were tested including luminance gratings (L) and second-order modulations of contrast (CM), spatial frequency (FM) and orientation (OM). Observers were asked to compare the apparent height of gratings with that of either filled, unmodulated rectangles or unfilled rectangles. Rectangles were presented alone or as part of a step with a highlighter. In some conditions highlighters matched the properties of the grating; in others or not. In one critical experiment, the HVI was compared for steps with highlighters that were separated from the riser by a thin line and those where the risers and highlighters were continuous. All gratings except FM appeared taller when presented in the step configuration with a continuous, matching highlighter. This effect was greatly reduced when a thin line separated the grating from the highlighter and abolished for mis-matched highlighters and risers. In the rectangle conditions, all cues appeared taller than blank rectangles and L and CM appeared taller than filled-unmodulated rectangles. In conclusion, second-order cues may be useful for inducing the HVI onto steps. However, the ability of vertical stripes and edge-highlighters to accentuate perceived step height may be due to aggregation of the highlighter into the grating rather than the normal horizontal-vertical illusion.

Macaque V1 responses to 2nd-order contrast-modulated stimuli and the possible subcortical and cortical contributions

Natural images comprise contours and boundaries defined by 1st-order luminance-modulated (LM) cues that are readily encoded by V1 neurons, and 2nd-order contrast-modulated (CM) cues that carry local, but not over-the-space, luminance changes. The neurophysiological foundations for CM processing remain unsolved. Here we used two-photon calcium imaging to demonstrate that V1 superficial-layer neurons respond to both LM and CM gratings in awake, fixating, macaques, with overall LM responses stronger than CM responses. Furthermore, adaptation experiments revealed that LM responses were similarly suppressed by LM and CM adaptation, with moderately larger effects by iso-orientation adaptation than by orthogonal adaptation, suggesting that LM and CM orientation responses likely share a strong orientation-non-selective subcortical origin. In contrast, CM responses were substantially more suppressed by iso-orientation than by orthogonal LM and CM adaptation, likely suggesting stronger orientation-specific intracortical influences for CM responses than for LM responses, besides shared orientation-non-selective subcortical influences. These results thus may indicate a subcortical-to-V1 filter-rectify-filter mechanism for CM processing: Local luminance changes in CM stimuli are initially encoded by orientation-non-selective subcortical neurons, and the outputs are half-wave rectified, and then summed by V1 neurons to signal CM orientation, which may be further substantially refined by intracortical influences.

Synergy of spatial frequency and orientation bandwidth in texture segregation

Defining target textures by increased bandwidths in spatial frequency and orientation, we observed strong cue combination effects in a combined texture figure detection and discrimination task. Performance for double-cue targets was better than predicted by independent processing of either cue and even better than predicted from linear cue integration. Application of a texture-processing model revealed that the oversummative cue combination effect is captured by calculating a low-level summary statistic (\(\Delta CE_m\)), which describes the differential contrast energy to target and reference textures, from multiple scales and orientations, and integrating this statistic across channels with a winner-take-all rule. Modeling detection performance using a signal detection theory framework showed that the observers' sensitivity to single-cue and double-cue texture targets, measured in \(d^{\prime }\) units, could be reproduced with plausible settings for filter and noise parameters. These results challenge models assuming separate channeling of elementary features and their later integration, since oversummative cue combination effects appear as an inherent property of local energy mechanisms, at least for spatial frequency and orientation bandwidth-modulated textures.

Integration of contours defined by second-order contrast-modulation of texture

Boundaries in the visual world can be defined by changes in luminance and texture in the input image. A "contour integration" process joins together local changes into percepts of lines or edges. A previous study tested the integration of contours defined by second-order contrast-modulation. Their contours were placed in a background of random wavelets. Participants performed near chance. We re-visited second-order contour integration with a different task. Participants distinguished contours with "good continuation" from distractors. We measured thresholds in different amounts of external orientation or position noise. This gave two noise-masking functions. We also measured thresholds for contours with a baseline curvature to assess performance with more curvy targets. Our participants were able to discriminate the good continuation of second-order contours. Thresholds were higher than for first-order contours. In our modelling, we found this was due to multiple factors. There was a doubling of equivalent internal noise between first- and second-order contour integration. There was also a reduction in efficiency. The efficiency difference was only significant in our orientation noise condition. For both first- and second-order stimuli, participants were also able to perform our task with more curved contours. We conclude that humans can integrate second-order contours, even when they are curved. There is however reduced performance compared to first-order contours. We find both an impaired input to the integrating mechanism, and reduced efficiency seem responsible. Second-order contour integration may be more affected by the noise background used in the previous study. Difficulty segregating that background may explain their result.

Contingent adaptation in masking and surround suppression

Adaptation is the process that changes a neuron's response based on recent inputs. In the traditional model, a neuron's state of adaptation depends on the recent input to that neuron alone, whereas in a recently introduced model (Hebbian normalization), adaptation depends on the structure of neural correlated firing. In particular, increased response products between pairs of neurons leads to increased mutual suppression. We test a psychophysical prediction of this model: adaptation should depend on 2nd-order statistics of input stimuli. That is, if two stimuli excite two distinct sub-populations of neurons, then presenting those stimuli simultaneously during adaptation should strengthen mutual suppression between those subpopulations. We confirm this prediction in two experiments. In the first, pairing two gratings synchronously during adaptation (i.e., a plaid) rather than asynchronously (interleaving the two gratings in time) leads to increased effectiveness of one pattern for masking the other. In the second, pairing the gratings in a center-surround configuration results in reduced apparent contrast for the central grating when paired with the same surround (as compared with a condition in which the central grating appears with a different surround at test than during adaptation). These results are consistent with the prediction that an increase in response covariance leads to greater mutual suppression between neurons. This effect is detectable both at threshold (masking) and well above threshold (apparent contrast).

Second-order visual sensitivity in the aging population

Most visual and cognitive functions are affected by aging over the lifespan. In this study, our aim was to investigate the loss in sensitivity to different classes of second-order stimuli-a class of stimuli supposed to be mainly processed in extrastriate cortex-in the aging population. These stimuli will then allow one to identify specific cortical deficit independently of visibility losses in upstream parts of the visual pathway. For this purpose, we measured the sensitivity to first-order stimuli and second-order stimuli: orientation-modulated, motion-modulated or contrast-modulated as a function of spatial frequency in 50 aged participants. Overall, we observed a sensitivity loss for all classes of stimuli, but this loss differentially affects the three classes of second-order stimuli tested. It involves all modulation spatial frequencies in the case of motion modulation, but just high modulation spatial frequencies in the case of contrast- and orientation modulations. These observations imply that aging selectively affects the sensitivity to second-order stimuli depending on their type. Since there is evidence that these different second-order stimuli are processed in different regions of extrastriate cortex, this result may suggest that some visual cortical areas are more susceptible to aging effects than others.

Spatial summation of broadband contrast

Spatial summation of luminance contrast signals has historically been psychophysically measured with stimuli isolated in spatial frequency (i.e., narrowband). Here, we revisit the study of spatial summation with noise patterns that contain the naturalistic 1/fα distribution of contrast across spatial frequency. We measured amplitude spectrum slope (α) discrimination thresholds and verified if sensitivity to α improved according to stimulus size. Discrimination thresholds did decrease with an increase in stimulus size. These data were modeled with a summation model originally designed for narrowband stimuli (i.e., single detecting channel; Baker & Meese, 2011; Meese & Baker, 2011) that we modified to include summation across multiple-differently tuned-spatial frequency channels. To fit our data, contrast gain control weights had to be inversely related to spatial frequency (1/f); thus low spatial frequencies received significantly more divisive inhibition than higher spatial frequencies, which is a similar finding to previous models of broadband contrast perception (Haun & Essock, 2010; Haun & Peli, 2013). We found summation across spatial frequency channels to occur prior to summation across space, channel summation was near linear and summation across space was nonlinear. Our analysis demonstrates that classical psychophysical models can be adapted to computationally define visual mechanisms under broadband visual input, with the adapted models offering novel insight on the integration of signals across channels and space.

Modeling second-order boundary perception: A machine learning approach

Visual pattern detection and discrimination are essential first steps for scene analysis. Numerous human psychophysical studies have modeled visual pattern detection and discrimination by estimating linear templates for classifying noisy stimuli defined by spatial variations in pixel intensities. However, such methods are poorly suited to understanding sensory processing mechanisms for complex visual stimuli such as second-order boundaries defined by spatial differences in contrast or texture. We introduce a novel machine learning framework for modeling human perception of second-order visual stimuli, using image-computable hierarchical neural network models fit directly to psychophysical trial data. This framework is applied to modeling visual processing of boundaries defined by differences in the contrast of a carrier texture pattern, in two different psychophysical tasks: (1) boundary orientation identification, and (2) fine orientation discrimination. Cross-validation analysis is employed to optimize model hyper-parameters, and demonstrate that these models are able to accurately predict human performance on novel stimulus sets not used for fitting model parameters. We find that, like the ideal observer, human observers take a region-based approach to the orientation identification task, while taking an edge-based approach to the fine orientation discrimination task. How observers integrate contrast modulation across orientation channels is investigated by fitting psychophysical data with two models representing competing hypotheses, revealing a preference for a model which combines multiple orientations at the earliest possible stage. Our results suggest that this machine learning approach has much potential to advance the study of second-order visual processing, and we outline future steps towards generalizing the method to modeling visual segmentation of natural texture boundaries. This study demonstrates how machine learning methodology can be fruitfully applied to psychophysical studies of second-order visual processing.

Many naturally occurring visual boundaries are defined by spatial differences in features other than luminance, for example by differences in texture or contrast. Quantitative models of such “second-order” boundary perception cannot be estimated using the standard regression techniques (known as “classification images”) commonly applied to “first-order”, luminance-defined stimuli. Here we present a novel machine learning approach to modeling second-order boundary perception using hierarchical neural networks. In contrast to previous quantitative studies of second-order boundary perception, we directly estimate network model parameters using psychophysical trial data. We demonstrate that our method can reveal different spatial summation strategies that human observers utilize for different kinds of second-order boundary perception tasks, and can be used to compare competing hypotheses of how contrast modulation is integrated across orientation channels. We outline extensions of the methodology to other kinds of second-order boundaries, including those in natural images.

Image segmentation driven by elements of form

While luminance, contrast, orientation, and terminators are well-established features that are extracted in early visual processing and support the parsing of an image into its component regions, the role of more complex features, such as closure and convexity, is less clear. A main barrier in understanding the roles of such features is that manipulating their occurrence typically entails changes in the occurrence of more elementary features as well. To address this problem, we developed a set of synthetic visual textures, constructed by replacing the binary coloring of standard maximum-entropy textures with tokens (tiles) containing curved or angled elements. The tokens were designed so that there were no discontinuities at their edges, and so that changing the correlation structure of the underlying binary texture changed the shapes that were produced. The resulting textures were then used in psychophysical studies, demonstrating that the resulting feature differences sufficed to drive segmentation. However, in contrast to previous findings for lower-level features, sensitivities to increases and decreases of feature occurrence were unequal. Moreover, the texture-segregation response depended on the kind of token (curved vs. angular, filled-in vs. outlined), and not just on the correlation structure. Analysis of this dependence indicated that simple closed contours and convex elements suffice to drive image segmentation, in the absence of changes in lower-level cues.

Attention alters spatial resolution by modulating second-order processing

Endogenous and exogenous visuospatial attention both alter spatial resolution, but they operate via distinct mechanisms. In texture segmentation tasks, exogenous attention inflexibly increases resolution even when detrimental for the task at hand and does so by modulating second-order processing. Endogenous attention is more flexible and modulates resolution to benefit performance according to task demands, but it is unknown whether it also operates at the second-order level. To answer this question, we measured performance on a second-order texture segmentation task while independently manipulating endogenous and exogenous attention. Observers discriminated a second-order texture target at several eccentricities. We found that endogenous attention improved performance uniformly across eccentricity, suggesting a flexible mechanism that can increase or decrease resolution based on task demands. In contrast, exogenous attention improved performance in the periphery but impaired it at central retinal locations, consistent with an inflexible resolution enhancement. Our results reveal that endogenous and exogenous attention both alter spatial resolution by differentially modulating second-order processing.

Textures as Probes of Visual Processing

Visual textures are a class of stimuli with properties that make them well suited for addressing general questions about visual function at the levels of behavior and neural mechanism. They have structure across multiple spatial scales, they put the focus on the inferential nature of visual processing, and they help bridge the gap between stimuli that are analytically convenient and the complex, naturalistic stimuli that have the greatest biological relevance. Key questions that are well suited for analysis via visual textures include the nature and structure of perceptual spaces, modulation of early visual processing by task, and the transformation of sensory stimuli into patterns of population activity that are relevant to perception.

The Effect of Spatial-Frequency Filtering on the Visual Processing of Global Structure

In three experiments we measured reaction times (RTs) and error rates in identifying the global structure of spatially filtered stimuli whose spatial-frequency content was selected by means of three types of 2-D isotropic filters (Butterworth of order 2, Butterworth of order 10, and a filters with total or partial Gaussian spectral profile). In each experiment, low-pass (LP), band-pass (BP), and high-pass (HP) filtered stimuli, with nine centre or cut-off spatial frequencies, were used. Irrespective of the type of filter, the experimental results showed that: (a) RTs to stimuli with low spatial frequencies were shorter than those to stimuli with medium or high spatial frequencies, (b) RTs to LP filtered stimuli were nearly constant, but they increased in a non-monotonic way with the filter centre spatial frequency in BP filtered stimuli and with the filter cut-off frequency in HP filtered stimuli, and (c) the identification of the global pattern occurred with all visible stimuli used, including BP and HP images without low spatial frequencies. To remove the possible influence of the energy, a fourth experiment was conducted with Gaussian filtered stimuli of equal contrast power ( crms = 0.065). Similar results to those described above were found for stimuli with spatial-frequency content higher than 2 cycles deg−1. A model of isotropic first-order visual channels collecting the stimulus spectral energy in all orientations explains the RT data. A subsequent second-order nonlinear amplitude demodulation process, applied to the output of the most energetic first-order channel, could explain the perception of global structure of each spatially filtered stimulus, including images lacking low spatial frequencies.

Specificity of brain reactions to second-order visual stimuli

The second-order visual mechanisms perform the operation of integrating the spatially distributed local visual information. Their organization is traditionally considered within the framework of the filter-rectify-filter model. These are the second-order filters that provide the ability to detect texture gradients. However, the question of the mechanisms' selectivity to the modulation dimension remains open. The aim of this investigation is to answer the above question by using visual evoked potentials (VEPs). Stimuli were textures consisting of staggered Gabor patches. The base texture was nonmodulated (NM). Three other textures represented the base texture which was sinusoidally modulated in different dimensions: contrast, orientation, or spatial frequency. EEG was recorded with 20 electrodes. VEPs of 500 ms duration were obtained for each of the four textures. After that, VEP to the NM texture was subtracted from VEP to each modulated texture. As a result, three different waves (d-waves) were obtained for each electrode site. Each d-wave was then averaged across all the 48 observers. The revealed d-waves have a latency of about 200 ms and, in our opinion, reflect the second-order filters reactivation through the feedback connection. The d-waves for different modulation dimensions were compared with each other in time, amplitude, topography, and localization of the sources of activity that causes the d-wave (with sLORETA). We proceeded from the assumption that the d-wave (its first component) represents functioning of the second-order visual mechanisms and activity changes at the following processing stages. It was found that the d-waves for different modulation dimensions significantly differ in all parameters. The obtained results indicate that the spatial modulations of different texture parameters caused specific changes in the brain activity, which could be evidence supporting the specificity of the second-order visual mechanisms to modulation dimension.

A perceptual space of local image statistics

Local image statistics are important for visual analysis of textures, surfaces, and form. There are many kinds of local statistics, including those that capture luminance distributions, spatial contrast, oriented segments, and corners. While sensitivity to each of these kinds of statistics have been well-studied, much less is known about visual processing when multiple kinds of statistics are relevant, in large part because the dimensionality of the problem is high and different kinds of statistics interact. To approach this problem, we focused on binary images on a square lattice - a reduced set of stimuli which nevertheless taps many kinds of local statistics. In this 10-parameter space, we determined psychophysical thresholds to each kind of statistic (16 observers) and all of their pairwise combinations (4 observers). Sensitivities and isodiscrimination contours were consistent across observers. Isodiscrimination contours were elliptical, implying a quadratic interaction rule, which in turn determined ellipsoidal isodiscrimination surfaces in the full 10-dimensional space, and made predictions for sensitivities to complex combinations of statistics. These predictions, including the prediction of a combination of statistics that was metameric to random, were verified experimentally. Finally, check size had only a mild effect on sensitivities over the range from 2.8 to 14min, but sensitivities to second- and higher-order statistics was substantially lower at 1.4min. In sum, local image statistics form a perceptual space that is highly stereotyped across observers, in which different kinds of statistics interact according to simple rules.

Visual perception of order-disorder transition

Our experience with the natural world, as composed of ordered entities, implies that perception captures relationships between image parts. For instance, regularities in the visual scene are rapidly identified by our visual system. Defining the regularities that govern perception is a basic, unresolved issue in neuroscience. Mathematically, perfect regularities are represented by symmetry (perfect order). The transition from ordered configurations to completely random ones has been extensively studied in statistical physics, where the amount of order is characterized by a symmetry-specific order parameter. Here we applied tools from statistical physics to study order detection in humans. Different sets of visual textures, parameterized by the thermodynamic temperature in the Boltzmann distribution, were designed. We investigated how much order is required in a visual texture for it to be discriminated from random noise. The performance of human observers was compared to Ideal and Order observers (based on the order parameter). The results indicated a high consistency in performance across human observers, much below that of the Ideal observer, but well-approximated by the Order observer. Overall, we provide a novel quantitative paradigm to address order perception. Our findings, based on this paradigm, suggest that the statistical physics formalism of order captures regularities to which the human visual system is sensitive. An additional analysis revealed that some order perception properties are captured by traditional texture discrimination models according to which discrimination is based on integrated energy within maps of oriented linear filters.

Surround suppression supports second-order feature encoding by macaque V1 and V2 neurons

Single neurons in areas V1 and V2 of macaque visual cortex respond selectively to luminance-modulated stimuli. These responses are often influenced by context, for example when stimuli extend outside the classical receptive field (CRF). These contextual phenomena, observed in many sensory areas, reflect a fundamental cortical computation and may inform perception by signaling second-order visual features which are defined by spatial relationships of contrast, orientation and spatial frequency. In the anesthetized, paralyzed macaque, we measured single-unit responses to a drifting preferred sinusoidal grating; low spatial frequency sinusoidal contrast modulations were applied to the grating, creating contrast-modulated, second-order forms. Most neurons responded selectively to the orientation of the contrast modulation of the preferred grating and were therefore second-order orientation-selective. Second-order selectivity was created by the asymmetric spatial organization of the excitatory CRF and suppressive extraclassical surround. We modeled these receptive field subregions using spatial Gaussians, sensitive to the modulation of contrast (not luminance) of the preferred carrier grating, that summed linearly and were capable of recovering asymmetrical receptive field organizations. Our modeling suggests that second-order selectivity arises both from elongated excitatory CRFs, asymmetrically organized extraclassical surround suppression, or both. We validated the model by successfully testing its predictions against conventional surround suppression measurements and spike-triggered analysis of second-order form responses. Psychophysical adaptation measurements on human observers revealed a pattern of second-order form selectivity consistent with neural response patterns. We therefore propose that cortical cells in primates do double duty, providing signals about both first- and second-order forms.

Pooling of first-order inputs in second-order vision

The processing of texture patterns has been characterized by a model that first filters the image to isolate one texture component, then applies a rectifying nonlinearity that converts texture variation into intensity variation, and finally processes the resulting pattern with mechanisms similar to those used in processing luminance-defined images (spatial-frequency- and orientation-tuned filters). This model, known as FRF for filter rectify filter, has the appeal of explaining sensitivity to second-order patterns in terms of mechanisms known to exist for processing first-order patterns. This model implies an unexpected interaction between the first and second stages of filtering; if the first-stage filter consists of narrowband mechanisms tuned to detect the carrier texture, then sensitivity to high-frequency texture modulations should be much lower than is observed in humans. We propose that the human visual system must pool over first-order channels tuned to a wide range of spatial frequencies and orientations to achieve texture demodulation, and provide psychophysical evidence for pooling in a cross-carrier adaptation experiment and in an experiment that measures modulation contrast sensitivity at very low first-order contrast.

Texture sparseness, but not local phase structure, impairs second-order segmentation

Texture boundary segmentation is typically thought to reflect a comparison of differences in Fourier energy (i.e. low-order texture statistics) on either side of a boundary. However in a previous study (Arsenault, Yoonessi, & Baker, 2011) we showed that the distribution of energy within a natural texture (i.e. its higher-order statistical structure) also influences segmentation of contrast boundaries. Here we examine the influence of specific higher-order texture statistics on segmentation of contrast- and orientation-defined boundaries. Using naturalistic synthetic textures to manipulate the sparseness, global phase structure, and local phase alignments of carrier textures, we measure segmentation thresholds based on forced-choice judgments of boundary orientation. We find a similar pattern of results for both contrast and orientation boundaries: (1) randomizing all structure by globally phase scrambling the texture reduces segmentation thresholds substantially, (2) decreasing sparseness also reduces thresholds, and (3) removing local phase alignments has little or no effect on segmentation thresholds. We show that a two-stage filter model with an intermediate compressive nonlinearity and expansive output nonlinearity can account for these data using synthetic textures. Furthermore, the model parameter fits obtained using synthetic textures also predict the segmentation thresholds presented in Arsenault, Yoonessi, and Baker (2011) for natural and phase-scrambled natural texture carriers.

Inconsistent channel bandwidth estimates suggest winner-take-all nonlinearity in second-order vision

The processing of texture patterns has been characterized by a model that postulates a first-stage linear filter to highlight a component texture, a pointwise rectification stage to convert contrast for the highlighted texture into mean response strength, followed by a second-stage linear filter to detect the texture-defined pattern. We estimated the spatial-frequency bandwidth of the second-stage filter mediating orientation discrimination of orientation-modulated second-order gratings by measuring threshold elevation in the presence of filtered noise added to the modulation signal. This experiment yielded no evidence for frequency tuning. A second experiment, in which subjects had to detect similar second-order gratings while judging their modulation frequency, produced bandwidth estimates of 1-1.5 octaves, similar to estimated bandwidths of first-order channels. We propose that an additional dominant-response-selection nonlinearity can account for these apparently contradictory results.

Differential effects of exogenous and endogenous attention on second-order texture contrast sensitivity

The visual system can use a rich variety of contours to segment visual scenes into distinct perceptually coherent regions. However, successfully segmenting an image is a computationally expensive process. Previously we have shown that exogenous attention--the more automatic, stimulus-driven component of spatial attention--helps extract contours by enhancing contrast sensitivity for second-order, texture-defined patterns at the attended location, while reducing sensitivity at unattended locations, relative to a neutral condition. Interestingly, the effects of exogenous attention depended on the second-order spatial frequency of the stimulus. At parafoveal locations, attention enhanced second-order contrast sensitivity to relatively high, but not to low second-order spatial frequencies. In the present study we investigated whether endogenous attention-the more voluntary, conceptually-driven component of spatial attention--affects second-order contrast sensitivity, and if so, whether its effects are similar to those of exogenous attention. To that end, we compared the effects of exogenous and endogenous attention on the sensitivity to second-order, orientation-defined, texture patterns of either high or low second-order spatial frequencies. The results show that, like exogenous attention, endogenous attention enhances second-order contrast sensitivity at the attended location and reduces it at unattended locations. However, whereas the effects of exogenous attention are a function of the second-order spatial frequency content, endogenous attention affected second-order contrast sensitivity independent of the second-order spatial frequency content. This finding supports the notion that both exogenous and endogenous attention can affect second-order contrast sensitivity, but that endogenous attention is more flexible, benefitting performance under different conditions.

Responses to second-order texture modulations undergo surround suppression

First-order (contrast) surround suppression has been well characterized both psychophysically and physiologically,but relatively little is known as to whether the perception of second-order visual stimuli exhibits analogous center–surround interactions. Second-order surround suppression was characterized by requiring subjects to detect second-order modulation in stimuli presented alone or embedded in a surround.Both contrast- (CM) and orientation-modulated (OM) stimuli were used. For most subjects and both OM and CM stimuli, second-order surrounds caused thresholds to be higher, indicative of second-order suppression. For CM stimuli, suppression was orientation-specific, i.e., higher thresholds for parallel than for orthogonal surrounds. However, the evidence for orientation specificity of suppression for OM stimuli was weaker. These results suggest that normalization, leading to surround suppression, operates at multiple stages in cortical processing.

Neuronal responses to texture-defined form in macaque visual area V2

Human and macaque observers can detect and discriminate visual forms defined by differences in texture. The neurophysiological correlates of visual texture perception are not well understood and have not been studied extensively at the single-neuron level in the primate brain. We used a novel family of texture patterns to measure the selectivity of neurons in extrastriate cortical area V2 of the macaque (Macaca nemestrina, Macaca fascicularis) for the orientation of texture-defined form, and to distinguish responses to luminance- and texture-defined form. Most V2 cells were selective for the orientation of luminance-defined form; they signaled the orientation of the component gratings that made up the texture patterns but not the overall pattern orientation. In some cells, these luminance responses were modulated by the direction or orientation of the texture envelope, suggesting an interaction of luminance and texture signals. We found little evidence for a "cue-invariant" representation in monkey V2. Few cells showed selectivity for the orientation of texture-defined form; they signaled the orientation of the texture patterns and not that of the component gratings. Small datasets recorded in monkey V1 and cat area 18 showed qualitatively similar patterns of results. Consistent with human functional imaging studies, our findings suggest that signals related to texture-defined form in primate cortex are most salient in areas downstream of V2. V2 may still provide the foundation for texture perception, through the interaction of luminance- and texture-based signals.

Orientation gradient detection exhibits variable coupling between first- and second-stage filtering mechanisms

We investigated sensitivity to orientation modulation using visual stimuli with bandpass filtered noise carriers. We characterized the relationship between the spatial parameters of the modulator and the carrier using a 2-AFC detection task. The relationship between these two parameters is potentially informative of the underlying coupling between first- and second-stage filtering mechanisms, which, in turn, may bear on the interrelationship between striate and extrastriate cortical processing. Our previous experiments on analogous motion stimuli found an optimum sensitivity when the ratio of the carrier and modulator spatial frequency parameters (r) was approximately ten. The current results do not exhibit an optimum sensitivity at a given value of the ratio r. Previous experiments involving second-order modulation sensitivity show an inconsistent range of estimates of optimum sensitivity at values of r between 5 and 50. Our results, using a complementary approach, confirm these discrepancies, demonstrating that the coupling between carrier and modulator frequency parameters depends on a number of stimulus-specific factors, such as contrast sensitivity, stimulus eccentricity, and absolute values of the carrier and modulator spatial frequency parameters. We show that these observations are true for a stimulus limited in eccentricity and that this orientation-modulated stimulus does not exhibit scale invariance. Such processing can not be modeled by a generic filter-rectify-filter model.

Human primary visual cortex (V1) is selective for second-order spatial frequency

A variety of cues can differentiate objects from their surrounds. These include "first-order" cues such as luminance modulations and "second-order" cues involving modulations of orientation and contrast. Human sensitivity to first-order modulations is well described by a computational model involving spatially localized filters that are selective for orientation and spatial frequency (SF). It is widely held that first-order modulations are represented by the firing rates of simple and complex cells ("first-order" neurons) in primary visual cortex (V1) that, likewise, have spatially localized receptive fields that are selective for orientation- and SF. Human sensitivity to second-order modulations is well described by a filter-rectify-filter (FRF) model, with first- and second-order filters selective for orientation and SF. However, little is known about how neuronal activity in visual cortex represents second-order modulations. We tested the FRF model by using an functional (f)MRI-adaptation protocol to characterize the selectivity of activity in visual cortex to second-order, orientation-defined gratings of two different SFs. fMRI responses throughout early visual cortex exhibited selective adaptation to these stimuli. The low-SF grating was a more effective adapter than the high-SF grating, incompatible with the FRF model. To explain the results, we extended the FRF model by incorporating normalization, yielding a filter-rectify-normalize-filter model, in which normalization enhances selectivity for second-order SF but only for low spatial frequencies. We conclude that neurons in human visual cortex are selective for second-order SF, that normalization (surround suppression) contributes to this selectivity, and that the selectivity in higher visual areas is simply fed forward from V1.

Exogenous attention enhances 2nd-order contrast sensitivity

Natural scenes contain a rich variety of contours that the visual system extracts to segregate the retinal image into perceptually coherent regions. Covert spatial attention helps extract contours by enhancing contrast sensitivity for 1st-order, luminance-defined patterns at attended locations, while reducing sensitivity at unattended locations, relative to neutral attention allocation. However, humans are also sensitive to 2nd-order patterns such as spatial variations of texture, which are predominant in natural scenes and cannot be detected by linear mechanisms. We assess whether and how exogenous attention--the involuntary and transient capture of spatial attention--affects the contrast sensitivity of channels sensitive to 2nd-order, texture-defined patterns. Using 2nd-order, texture-defined stimuli, we demonstrate that exogenous attention increases 2nd-order contrast sensitivity at the attended location, while decreasing it at unattended locations, relative to a neutral condition. By manipulating both 1st- and 2nd-order spatial frequency, we find that the effects of attention depend both on 2nd-order spatial frequency of the stimulus and the observer's 2nd-order spatial resolution at the target location. At parafoveal locations, attention enhances 2nd-order contrast sensitivity to high, but not to low 2nd-order spatial frequencies; at peripheral locations attention also enhances sensitivity to low 2nd-order spatial frequencies. Control experiments rule out the possibility that these effects might be due to an increase in contrast sensitivity at the 1st-order stage of visual processing. Thus, exogenous attention affects 2nd-order contrast sensitivity at both attended and unattended locations.

Beyond multiple pattern analyzers modeled as linear filters (as classical V1 simple cells): useful additions of the last 25 years

This review briefly discusses processes that have been suggested in the last 25 years as important to the intermediate stages of visual processing of patterns. Five categories of processes are presented: (1) Higher-order processes including FRF structures; (2) Divisive contrast nonlinearities including contrast normalization; (3) Subtractive contrast nonlinearities including contrast comparison; (4) Non-classical receptive fields (surround suppression, cross-orientation inhibition); (5) Contour integration.

Orientation selectivity of motion-boundary responses in human visual cortex

Motion boundaries (local changes in visual motion direction) arise naturally when objects move relative to an observer. In human visual cortex, neuroimaging studies have identified a region (the kinetic occipital area [KO]) that responds more strongly to motion-boundary stimuli than to transparent-motion stimuli. However, some functional magnetic resonance imaging (fMRI) studies suggest that KO may encompass multiple visual areas and single-unit studies in macaque visual cortex have identified neurons selective for motion-boundary orientation in areas V2, V3, and V4, implying that motion-boundary selectivity may not be restricted to a single area. It is not known whether fMRI responses to motion boundaries are selective for motion-boundary orientation, as would be expected if these responses reflected the population activity of motion-boundary-selective neurons. We used an event-related fMRI adaptation protocol to measure orientation-selective responses to motion boundaries in human visual cortex. On each trial, we measured the response to a probe stimulus presented after an adapter stimulus (a vertical or horizontal motion-boundary grating). The probe stimulus was either a motion-boundary grating oriented parallel or orthogonal to the adapter stimulus or a transparent-motion stimulus. Orientation-selective adaptation for motion boundaries--smaller responses for trials in which test and adapter stimuli were parallel to each other--was observed in multiple extrastriate visual areas. The strongest adaptation, relative to the unadapted responses, was found in V3A, V3B, LO1, LO2, and V7. Most of the visual areas that exhibited orientation-selective adaptation in our data also showed response preference for motion boundaries over transparent motion, indicating that most of the human visual areas previously shown to respond to motion boundaries are also selective for motion-boundary orientation. These results suggest that neurons selective for motion-boundary orientation are distributed across multiple human visual cortical areas and argue against the existence of a single region or area specialized for motion-boundary processing.

Development of sensitivity to visual texture modulation in macaque monkeys

In human and non-human primates, higher form vision matures substantially later than spatial acuity and contrast sensitivity, as revealed by performance on such tasks as figure-ground segregation and contour integration. Our goal was to understand whether delayed maturation on these tasks was intrinsically form-dependent or, rather, related to the nature of spatial integration necessary for extracting task-relevant cues. We used an intermediate-level form task that did not call for extensive spatial integration. We trained monkeys (6-201 weeks) to discriminate the orientation of pattern modulation in a two-alternative forced choice paradigm. We presented two families of form patterns, defined by texture or contrast variations, and luminance-defined patterns for comparison. Infant monkeys could discriminate texture- and contrast-defined form as early as 6 weeks; sensitivity improved up to 40 weeks. Surprisingly, sensitivity for texture- and contrast-defined form matured earlier than for luminance-defined form. These results suggest that intermediate-level form vision develops in concert with basic spatial vision rather than following sequentially. Comparison with earlier results reveals that different aspects of form vision develop over different time courses, with processes that depend on comparing local image content maturing earlier than those requiring "global" linking of multiple visual elements across a larger spatial extent.

Spatial summation of first-order and second-order motion in human vision

This study assessed spatial summation of first-order (luminance-defined) and second-order (contrast-defined) motion. Thresholds were measured for identifying the drift direction of 1c/deg., luminance-modulated and contrast-modulated dynamic noise drifting at temporal frequencies of 0.5, 2 and 8Hz. Image size varied from 0.125 degrees to 16 degrees . The effects of increasing image size on thresholds for luminance-modulated noise were also compared to those for luminance-defined gratings. In all cases, performance improved as image size increased. The rate at which performance improved with increasing image size was similar for all stimuli employed although the slopes corresponding to the initial improvement were steeper for first-order compared to second-order motion. The image sizes at which performance for first-order motion asymptote were larger than for second-order motion. In addition, findings showed that the minimum image size required to support reliable identification of the direction of moving stimuli is greater for second-order than first-order motion. Thus, although first-order and second-order motion processing have a number of properties in common, the visual system's sensitivity to each type of motion as a function of image size is quite different.

Scale dependence and channel switching in letter identification

Letters are broadband visual stimuli with information useful for discrimination over a wide range of spatial frequencies. Yet, recent evidence suggests that observers use only a single, fixed spatial-frequency channel to identify letters and that the scale of that channel, in units of letter size, is determined by the size of the letter (scale dependence). We report two letter-identification experiments using critical-band masking. With sufficiently high-amplitude, low- or high-pass masking noise, observers switched to a different range of spatial frequencies for the task. Thus, letter channels are not fixed for a given letter size. When an additional white-noise masker was added to the stimulus to flatten the contrast-sensitivity function, the letter channel used by the observer still depended on letter size, further supporting the hypothesis that letter identification is scale dependent.

Improved visual sensitivity during smooth pursuit eye movements: Temporal and spatial characteristics

Recently, we showed that contrast sensitivity for color and high–spatial frequency luminance stimuli is enhanced during smooth pursuit eye movements (Schütz et al., 2008). In this study, we investigated the enhancement over a wide range of temporal and spatial frequencies. In Experiment 1, we measured the temporal impulse response function (TIRF) for colored stimuli. The TIRF for pursuit and fixation differed mostly with respect to the gain but not with respect to the natural temporal frequency. Hence, the sensitivity enhancement seems to be rather independent of the temporal frequency of the stimuli. In Experiment 2, we measured the spatial contrast sensitivity function for luminance-defined Gabor patches with spatial frequencies ranging from 0.2 to 7 cpd. We found a sensitivity improvement during pursuit for spatial frequencies above 2–3 cpd. Between 0.5 and 3 cpd, sensitivity was impaired by smooth pursuit eye movements, but no consistent difference was observed below 0.5 cpd. The results of both experiments are consistent with an increased contrast gain of the parvocellular retinogeniculate pathway.

Visual image reconstruction from human brain activity using a combination of multiscale local image decoders

Perceptual experience consists of an enormous number of possible states. Previous fMRI studies have predicted a perceptual state by classifying brain activity into prespecified categories. Constraint-free visual image reconstruction is more challenging, as it is impractical to specify brain activity for all possible images. In this study, we reconstructed visual images by combining local image bases of multiple scales, whose contrasts were independently decoded from fMRI activity by automatically selecting relevant voxels and exploiting their correlated patterns. Binary-contrast, 10 x 10-patch images (2(100) possible states) were accurately reconstructed without any image prior on a single trial or volume basis by measuring brain activity only for several hundred random images. Reconstruction was also used to identify the presented image among millions of candidates. The results suggest that our approach provides an effective means to read out complex perceptual states from brain activity while discovering information representation in multivoxel patterns.

Single-band amplitude demodulation of Müller-Lyer illusion images

The perception of the Müller-Lyer illusion has previously been explained as a result of visual low band-pass spatial filtering, although, in fact, the illusion persists in band-pass and high-pass filtered images without visible low-spatial frequencies. A new theoretical framework suggests that our perceptual experience about the global spatial structure of an image corresponds to the amplitude modulation (AM) component (or its magnitude, also called envelope) of its AM-FM (alternatively, AM-PM) decomposition. Because demodulation is an ill-posed problem with a non-unique solution, two different AM-FM demodulation algorithms were applied here to estimate the envelope of images of Müller-Lyer illusion: the global and exact Daugman and Downing (1995) AMPM algorithm and the local and quasi-invertible Maragos and Bovik (1995) DESA. The images used in our analysis include the classic configuration of illusion in a variety of spatial and spatial frequency content conditions. In all cases, including those of images for which visual low-pass spatial filtering would be ineffective, the envelope estimated by single-band amplitude demodulation has physical distortions in the direction of perceived illusion. It is not plausible that either algorithm could be implemented by the human visual system. It is shown that the proposed second order visual model of pre-attentive segregation of textures (or "back-pocket" model) could recover the image envelope and, thus, explain the perception of this illusion even in Müller-Lyer images lacking low spatial frequencies.

Crowding between first- and second-order letter stimuli in normal foveal and peripheral vision

Evidence that the detection of first- and second-order visual stimuli is processed by separate pathways abounds. This study asked whether first- and second-order stimuli remain independent at the stage of processing where crowding occurs. We measured thresholds for identifying a first-order (luminance defined) or second-order (contrast defined) target letter in the presence of two second- or first-order flanking letters. For comparison, we also measured thresholds when the target and flanking letters were all first or second order. Contrast of the flankers was 1.6 times their respective contrast thresholds. Measurements were obtained at the fovea and 10 degrees in the lower visual field of four normally sighted observers. Two observers were also tested at 10 degrees nasal visual field. As expected, in both the fovea and periphery, the magnitude of crowding (threshold elevation) was maximal at the closest letter separation and decreased as letter separation increased. The magnitude of crowding was greater for second- than for first-order target letters, independent of the order type of flankers; however, the critical distance for crowding was similar for first- and second-order letters. Substantial crossover crowding occurred when the target and flanking letters were of different order type. Our finding of substantial interaction between first- and second-order stimuli suggests that the processing of these stimuli is not independent at the stage of processing at which crowding occurs.

Region- and edge-based configurational effects in texture segmentation

We have found a new configurational effect in texture segmentation. In addition to collinear facilitation at the edge, this effect results from contextual modulation within the texture-region, i.e. from texels not abutting the segmented edge. The largest facilitation was found when two conditions were fulfilled: (i) elements along the edge were parallel to the edge and collinear, (ii) elements in the texture-region were also collinear but non-parallel to the edge. We show that this facilitation occurs when there are groups of different orientation from the edge in the texture-region. We suggest two possible underlying mechanisms: either a region-based process that links collinear iso-oriented elements and locates the edge when the orientation changes, or else second-order filters tuned to orientation differences rather than orientation per se.

Separate first- and second-order processing is supported by spatial summation estimates at the fovea and eccentrically

We estimated spatial summation areas for the detection of luminance-modulated (LM) and contrast-modulated (CM) blobs at the fovea, 2.5, 5 and 10 deg eccentrically. Gaussian profiles were added or multiplied to binary white noise to create LM and CM blob stimuli and these were used to psychophysically estimate detection thresholds and spatial summation areas. The results reveal significantly larger summation areas for detecting CM than LM blobs across eccentricity. These differences are comparable to receptive field size estimates made in V1 and V2. They support the notion that separate spatial processing occurs for the detection of LM and CM stimuli.

Cue combination in a combined feature contrast detection and figure identification task

Target figures defined by feature contrast in spatial frequency, orientation or both cues had to be detected in Gabor random fields and their shape had to be identified in a dual task paradigm. Performance improved with increasing feature contrast and was strongly correlated among both tasks. Subjects performed significantly better with combined cues than with single cues. The improvement due to cue summation was stronger than predicted by the assumption of independent feature specific mechanisms, and increased with the performance level achieved with single cues until it was limited by ceiling effects. Further, cue summation was also strongly correlated among tasks: when there was benefit due to the additional cue in feature contrast detection, there was also benefit in figure identification. For the same performance level achieved with single cues, cue summation was generally larger in figure identification than in feature contrast detection, indicating more benefit when processes of shape and surface formation are involved. Our results suggest that cue combination improves spatial form completion and figure-ground segregation in noisy environments, and therefore leads to more stable object vision.

FMRI adaptation reveals separate mechanisms for first-order and second-order motion

A key unresolved debate in human vision concerns whether we have two different low-level mechanisms for encoding image motion. Separate neural mechanisms have been suggested for first-order (luminance modulation) and second-order (e.g., contrast modulation) motion in the retinal image but a single mechanism could handle both. Human functional magnetic resonance imaging (fMRI) has not so far convincingly revealed separate anatomical substrates. To examine whether two separate but co-localized mechanisms might exist, we used the technique of fast fMRI adaptation. We found direction-selective adaptation independently for each type of motion in the motion area V5/MT+ of the human brain. However, there was a total absence of cross-adaptation between first-order and second-order motion stimuli. This was true in both of the two subcomponents of MT+ (MT and MST) and similar results were found in V3A. This pattern of adaptation was consistent with psychophysical measurements of detection thresholds in similar stimulus sequences. The results provide strong evidence for separate neural populations that are responsible for detecting first- and second-order motion.

Noise masking reveals channels for second-order letters

We investigate the channels underlying identification of second-order letters using a critical-band masking paradigm. We find that observers use a single 1-1.5 octave-wide channel for this task. This channel's best spatial frequency (c/letter) did not change across different noise conditions (indicating the inability of observers to switch channels to improve signal-to-noise ratio) or across different letter sizes (indicating scale invariance), for a fixed carrier frequency (c/letter). However, the channel's best spatial frequency does change with stimulus carrier frequency (both in c/letter); one is proportional to the other. Following Majaj et al. (Majaj, N. J., Pelli, D. G., Kurshan, P., & Palomares, M. (2002). The role of spatial frequency channels in letter identification. Vision Research, 42, 1165-1184), we define "stroke frequency" as the line frequency (strokes/deg) in the luminance image. That is, for luminance-defined letters, stroke frequency is the number of lines (strokes) across each letter divided by letter width. For second-order letters, letter texture stroke frequency is the number of carrier cycles (luminance lines) within the letter ink area divided by the letter width. Unlike the nonlinear dependence found for first-order letters (implying scale-dependent processing), for second-order letters the channel frequency is half the letter texture stroke frequency (suggesting scale-invariant processing).

Orientation-selective adaptation to first- and second-order patterns in human visual cortex

Second-order textures-patterns that cannot be detected by mechanisms sensitive only to luminance changes-are ubiquitous in visual scenes, but the neuronal mechanisms mediating perception of such stimuli are not well understood. We used an adaptation protocol to measure neural activity in the human brain selective for the orientation of second-order textures. Functional MRI (fMRI) responses were measured in three subjects to presentations of first- and second-order probe gratings after adapting to a high-contrast first- or second-order grating that was either parallel or orthogonal to the probe gratings. First-order (LM) stimuli were generated by modulating the stimulus luminance. Second-order stimuli were generated by modulating the contrast (CM) or orientation (OM) of a first-order carrier. We used four combinations of adapter and probe stimuli: LM:LM, CM:CM, OM:OM, and LM:OM. The fourth condition tested for cross-modal adaptation with first-order adapter and second-order probe stimuli. Attention was diverted from the stimulus by a demanding task at fixation. Both first- and second-order stimuli elicited orientation-selective adaptation in multiple cortical visual areas, including V1, V2, V3, V3A/B, a newly identified visual area anterior to dorsal V3 that we have termed LO1, hV4, and VO1. For first-order stimuli (condition LM:LM), the adaptation was no larger in extrastriate areas than in V1, implying that the orientation-selective first-order (luminance) adaptation originated in V1. For second-order stimuli (conditions CM:CM and OM:OM), the magnitude of adaptation, relative to the absolute response magnitude, was significantly larger in VO1 (and for condition CM:CM, also in V3A/B and LO1) than in V1, suggesting that second-order stimulus orientation was extracted by additional processing after V1. There was little difference in the amplitude of adaptation between the second-order conditions. No consistent effect of adaptation was found in the cross-modal condition LM:OM, in agreement with psychophysical evidence for weak interactions between first- and second-order stimuli and computational models of separate mechanisms for first- and second-order visual processing.

Second-order spatial summation in amblyopia

Amblyopes show bilateral loss of sensitivity for second-order (contrast defined) stimuli that can be further suppressed by flanking second-order stimuli (whereas flanks facilitate sensitivity in normal observers). The suppressive flank effect in amblyopes might be explained by abnormal pooling of second-order contrast across visual space. In this study, we investigate whether amblyopes show abnormal second-order spatial summation by measuring contrast detection thresholds for 1c/deg modulations of random noise (stimuli 1-12 cycles) in amblyopic observers, strabismic observers with no visual acuity loss, and normal (control) observers. Non-control observers showed substantial bilateral loss of sensitivity relative to the control observers, as expected. However, all observers showed essentially equal second-order spatial summation: contrast detection threshold decreased with approximately the square root of the number of cycles, and then became independent of size at 6-8 cycles (similar asymptotes). We conclude that the pooling of second-order contrast across visual space is unaffected by amblyopia.

Nonlinear and higher-order approaches to the encoding of natural scenes

Linear operations can only partially exploit the statistical redundancies of natural scenes, and nonlinear operations are ubiquitous in visual cortex. However, neither the detailed function of the nonlinearities nor the higher-order image statistics are yet fully understood. We suggest that these complicated issues can not be tackled by one single approach, but require a range of methods, and the understanding of the crosslinks between the results. We consider three basic approaches: (i) State space descriptions can theoretically provide complete information about statistical properties and nonlinear operations, but their practical usage is confined to very low-dimensional settings. We discuss the use of representation-related state-space coordinates (multivariate wavelet statistics) and of basic nonlinear coordinate transformations of the state space (e.g., a polar transform). (ii) Indirect methods, like unsupervised learning in multi-layer networks, provide complete optimization results, but no direct information on the statistical properties, and no simple model structures. (iii) Approximation by lower-order terms of power-series expansions is a classical strategy that has not yet received broad attention. On the statistical side, this approximation amounts to cumulant functions and higher-order spectra (polyspectra), on the processing side to Volterra Wiener systems. In this context we suggest that an important concept for the understanding of natural scene statistics, of nonlinear neurons, and of biological pattern recognition can be found in AND-like combinations of frequency components. We investigate how the different approaches can be related to each other, how they can contribute to the understanding of cortical nonlinearities such as complex cells, cortical gain control, end-stopping and other extraclassical receptive field properties, and how we can obtain a nonlinear perspective on overcomplete representations and invariant coding in visual cortex.

Is there opponent-orientation coding in the second-order channels of pattern vision?

Is there opponency between orientation-selective processes in pattern perception, analogous to opponency between color mechanisms? Here we concentrate on possible opponency in second-order channels. We compare several possible second-order structures: SIGN-opponent-only channels in which there is no opponency between orientations (also called complex channels or filter-rectify-filter mechanisms); three structures we group under the name ORIENTATION-opponent; and finally BOTH-opponent channels which combine features of both SIGN-opponent-only and ORIENTATION-opponent channels but lead to predictions that are distinct from either of theirs. We measured observers' ability to segregate textures composed of checkerboard and striped arrangements of vertical and horizontal Gabor grating patches. The observers' performance was compared to model predictions from the alternative opponent structures. The experimental results are consistent with SIGN-opponent-only channels. The results rule out the ORIENTATION-opponent and BOTH-opponent structures. Further, when the models were expanded to include a contrast gain-control (inhibition among channels in a normalization network) the SIGN-opponent-only model was also able to explain a contrast-dependent effect we found, thus providing another piece of evidence that such normalization is an important process in human texture perception.

Cross-orientation summation in texture segregation

Human texture vision has been modeled as a filter-rectify-filter (FRF) process, in which '2nd-order' filters detect changes in the rectified outputs of luminance-based '1st-order' filters. This study tested the validity of the two basic assumptions of the standard FRF model, namely (a) that the 2nd-order filters are sensitive to spatial modulations in both contrast and orientation, and (b) that the 2nd-order filters are tuned to different 1st-order orientations. In the first experiment, we tested subthreshold summation between two orthogonal carrier orientations in detection of a texture region, which was defined by contrast modulations across regions in the two carrier orientations, while systematically varying the relative change magnitudes between the two orientations. The results showed that the detection thresholds were determined by spatial difference in the contrast integrated over the two orientations. Orientation difference did act as a segregation cue, but only when there was no differences in carrier contrast. This suggests that two mechanisms are involved in texture segregation; one that detects changes in luminance contrast and another that detects changes in orientation. To further analyze the latter mechanism, a second experiment measured cross-orientation summation in the detection of purely orientation-defined textures, using stimuli that were density modulations of two orientations presented among randomly-orientated distractors. Again, the relative modulation magnitudes between the two orientations was systematically varied. The results are consistent with the notions that (a) the dominant orientation is extracted from the 1st-order outputs before the 2nd-order process, and that (b) the 2nd-order, spatial comparison process integrates those dominant signals over different orientations.

Orientation opponency in human vision revealed by energy-frequency analysis

Studies of second-order visual processing have primarily been concerned with understanding the mechanisms for detecting spatiotemporal variations in such attributes as contrast, orientation, spatial frequency, etc. Here, we have examined the orientation characteristics of second-order processes using bandpass noise whose Fourier energy is sinusoidally modulated across orientation, rather than across space or time. Sensitivity for detecting orientation-energy modulations was measured as a function of modulation frequency. The sensitivity function was bandpass, with a pronounced peak at an orientation frequency of 4 cycles/pi. An inverse Fourier transform of the sensitivity function revealed a filter profile displaying a centre-surround antagonism across orientation, with an excitatory centre within 6-9 deg and inhibitory lobes at 15-20 deg from the filter's centre. The degree of centre-surround antagonism increased with stimulus size far beyond the spatial range of the first-order filters (more than 64 times the dominant spatial wavelength of the noise carrier). These results suggest that second-order processing involves 'orientation-opponent' channels that extract differences in first-order outputs across orientation over a wide area of the visual field.