A processing stream in mammalian visual cortex neurons for non-Fourier responses

Separate first- and second-order processing is supported by spatial summation estimates at the fovea and eccentrically

We estimated spatial summation areas for the detection of luminance-modulated (LM) and contrast-modulated (CM) blobs at the fovea, 2.5, 5 and 10 deg eccentrically. Gaussian profiles were added or multiplied to binary white noise to create LM and CM blob stimuli and these were used to psychophysically estimate detection thresholds and spatial summation areas. The results reveal significantly larger summation areas for detecting CM than LM blobs across eccentricity. These differences are comparable to receptive field size estimates made in V1 and V2. They support the notion that separate spatial processing occurs for the detection of LM and CM stimuli.

Contour discontinuities subserve two types of form analysis that underlie motion processing

Form analysis subserves motion processing in at least two ways: first, in terms of figural segmentation dedicated to solving the problem of figure-to-figure matching over time, and second, in terms of defining trackable features whose unambiguous motion signals can be generalized to ambiguously moving portions of an object. The former is a primarily ventral process involving the lateral occipital complex and also retinotopic areas such as V2 and V4, and the latter is a dorsal process involving V3A. Contour discontinuities, such as corners, deep concavities, maxima of positive curvature, junctions, and terminators, play a central role in both types of form analysis. Transformational apparent motion will be discussed in the context of figural segmentation and matching, and rotational motion in the context of trackable features. In both cases the analysis of form must proceed in parallel with the analysis of motion, in order to constrain the ongoing analysis of motion.

Processing of first-order motion in marmoset visual cortex is influenced by second-order motion

We measured the responses of single neurons in marmoset visual cortex (V1, V2, and the third visual complex) to moving first-order stimuli and to combined first- and second-order stimuli in order to determine whether first-order motion processing was influenced by second-order motion. Beat stimuli were made by summing two gratings of similar spatial frequency, one of which was static and the other was moving. The beat is the product of a moving sinusoidal carrier (first-order motion) and a moving low-frequency contrast envelope (second-order motion). We compared responses to moving first-order gratings alone with responses to beat patterns with first-order and second-order motion in the same direction as each other, or in opposite directions to each other in order to distinguish first-order and second-order direction-selective responses. In the majority (72%, 67/93) of cells (V1 73%, 45/62; V2 70%, 16/23; third visual complex 75%, 6/8), responses to first-order motion were significantly influenced by the addition of a second-order signal. The second-order envelope was more influential when moving in the opposite direction to the first-order stimulus, reducing first-order direction sensitivity in V1, V2, and the third visual complex. We interpret these results as showing that first-order motion processing through early visual cortex is not separate from second-order motion processing; suggesting that both motion signals are processed by the same system.

The cellular basis for parallel neural transmission of a high-frequency stimulus and its low-frequency envelope

Sensory stimuli often have rich temporal and spatial structure. One class of stimuli that are common to visual and auditory systems and, as we show, the electrosensory system are signals that contain power in a narrow range of temporal (or spatial) frequencies. Characteristic of this class of signals is a slower variation in their amplitude, otherwise known as an envelope. There is evidence suggesting that, in the visual cortex, both narrowband stimuli and their envelopes are coded for in separate and parallel streams. The implementation of this parallel transmission is not well understood at the cellular level. We have identified the cellular basis for the parallel transmission of signal and envelope in the electrosensory system: a two-cell network consisting of an interneuron connected to a pyramidal cell by means of a slow synapse. This circuit could, in principle, be implemented in the auditory or visual cortex by the previously identified biophysics of cortical interneurons.

Neural mechanisms mediating responses to abutting gratings: luminance edges vs. illusory contours

The discontinuities of phase-shifted abutting line gratings give rise to perception of an "illusory contour" (IC) along the line terminations. Neuronal responses to such ICs have been interpreted as evidence for a specialized visual mechanism, since such responses cannot be predicted from conventional linear receptive fields. However, when the spatial scale of the component gratings (carriers) is large compared to the neuron's luminance passband, these IC responses might be evoked simply by the luminance edges at the line terminations. Thus by presenting abutting gratings at a series of carrier spatial scales to cat A18 neurons, we were able to distinguish genuine nonlinear responses from those due to luminance edges. Around half of the neurons (both simple and complex types) showed a bimodal response pattern to abutting gratings: one peak at a low carrier spatial frequency range that overlapped with the luminance passband, and a second distinct peak at much higher frequencies beyond the neuron's grating resolution. For those bimodally responding neurons, the low-frequency responses were sensitive to carrier phase, but the high-frequency responses were phase-invariant. Thus the responses at low carrier spatial frequencies could be understood via a linear model, while the higher frequency responses represented genuine nonlinear IC processing. IC responsive neurons also demonstrated somewhat lower spatial preference to the periodic contours (envelopes) compared to gratings, but the optimal orientation and motion direction for both were quite similar. The nonlinear responses to ICs could be explained by the same energy mechanism underlying responses to second-order stimuli such as contrast-modulated gratings. Similar neuronal preferences for ICs and for gratings may contribute to the form-cue invariant perception of moving contours.

The role of human extra-striate visual areas V5/MT and V2/V3 in the perception of the direction of global motion: a transcranial magnetic stimulation study

Several published single case studies reveal a double dissociation between the effects of brain damage in separate extra-striate cortical visual areas on the perception of global visual motion defined by a difference in luminance (first-order motion) versus motion defined by a difference in contrast (second-order motion). In particular, the medial extrastriate cortical region V2/V3 seems to be crucial for the perception of first-order motion, but not for second-order, whereas a lateral and more anterior portion of the cortex close to the temporo-parieto-occipital junction (in the territory of the human motion area hV5/MT+) seems to be essential only for the perception of second-order motion. In order to test the hypothesis of a functional specialization of different visual areas for different types of motion, we applied repetitive transcranial magnetic stimulation (rTMS) unilaterally over areas V2/V3, V5/MT, or posterior parietal cortex (PPC) while subjects performed a 2AFC task with first- or second-order global motion displays in the contralateral visual field. Results showed a comparable disruption of the two types of motion, with both rTMS over V2/V3 or over MT/V5, and little or no effect with rTMS over PPC. The results suggest that either the previous psychophysical results with neurological patients are incorrect (highly unlikely) or that the lateral and medial regions are directly connected (as they are in macaque monkeys) such that stimulating one automatically affects the other, in this instance disruptively.

Second-order spatial frequency and orientation channels in human vision

We compared the number of spatial frequency and orientation mechanisms underlying first- versus second-order processing by measuring discrimination at detection threshold for first- and second-order Gabors to determine the smallest difference in spatial frequency and orientation that permits accurate discrimination at threshold. For second-order gratings, the number of channels is the same as for first-order gratings for spatial frequencies up to about 2 cpd; however, there are fewer second-order channels at higher spatial frequencies. In contrast, the number of labeled channels for orientation is the same for first- and second-order gratings. In conclusion, our findings provide evidence for distinct spatial frequency and orientation labeled detectors in second-order visual processing. We also show that, relative to first-order, there are fewer second-order channels processing higher spatial frequencies. This is consistent with a filter-rectify-filter scheme for second-order in which the second stage of filtering is at lower spatial frequencies.

Greater losses in sensitivity to second-order local motion than to first-order local motion after early visual deprivation in humans

We compared sensitivity to first-order versus second-order local motion in patients treated for dense central congenital cataracts in one or both eyes. Amplitude modulation thresholds were measured for discriminating the direction of motion of luminance-modulated (first-order) and contrast modulated (second-order) horizontal sine-wave gratings. Early visual deprivation, whether monocular or binocular, caused losses in sensitivity to both first- and second-order motion, with greater losses for second-order motion than for first-order motion. These findings are consistent with the hypothesis that the two types of motion are processed by different mechanisms and suggest that those mechanisms are differentially sensitive to early visual input.

Orientation-selective adaptation to first- and second-order patterns in human visual cortex

Second-order textures-patterns that cannot be detected by mechanisms sensitive only to luminance changes-are ubiquitous in visual scenes, but the neuronal mechanisms mediating perception of such stimuli are not well understood. We used an adaptation protocol to measure neural activity in the human brain selective for the orientation of second-order textures. Functional MRI (fMRI) responses were measured in three subjects to presentations of first- and second-order probe gratings after adapting to a high-contrast first- or second-order grating that was either parallel or orthogonal to the probe gratings. First-order (LM) stimuli were generated by modulating the stimulus luminance. Second-order stimuli were generated by modulating the contrast (CM) or orientation (OM) of a first-order carrier. We used four combinations of adapter and probe stimuli: LM:LM, CM:CM, OM:OM, and LM:OM. The fourth condition tested for cross-modal adaptation with first-order adapter and second-order probe stimuli. Attention was diverted from the stimulus by a demanding task at fixation. Both first- and second-order stimuli elicited orientation-selective adaptation in multiple cortical visual areas, including V1, V2, V3, V3A/B, a newly identified visual area anterior to dorsal V3 that we have termed LO1, hV4, and VO1. For first-order stimuli (condition LM:LM), the adaptation was no larger in extrastriate areas than in V1, implying that the orientation-selective first-order (luminance) adaptation originated in V1. For second-order stimuli (conditions CM:CM and OM:OM), the magnitude of adaptation, relative to the absolute response magnitude, was significantly larger in VO1 (and for condition CM:CM, also in V3A/B and LO1) than in V1, suggesting that second-order stimulus orientation was extracted by additional processing after V1. There was little difference in the amplitude of adaptation between the second-order conditions. No consistent effect of adaptation was found in the cross-modal condition LM:OM, in agreement with psychophysical evidence for weak interactions between first- and second-order stimuli and computational models of separate mechanisms for first- and second-order visual processing.

Is the direction of second-order, contrast-defined motion patterns visible to standard motion-energy detectors: a model answer?

Previous psychophysical studies (e.g., Smith & Ledgeway, 1997) have provided evidence that under some conditions, the detection of a particular class of stimuli (contrast-modulated static noise) widely employed to study second-order motion processing may be inadvertently based on encoding local imbalances in luminance motion energy. In particular when static noise composed of relatively large noise elements is used, direction-identification performance at threshold may actually be mediated by the same mechanisms that respond to first-order motion, due to the presence of persistent spatial clusters of noise elements of the same polarity. However, Benton and Johnson (1997) modeled the responses of conventional motion-energy detectors to contrast-modulated static noise patterns and found no evidence of any systematic directional biases in such stimuli when the mean opponent motion energy was used to quantify performance. In the present paper we sought to resolve this discrepancy and show that the precise manner in which computational models are implemented is crucial in determining their response to contrast-modulated, second-order motion patterns. In particular we demonstrate that by considering the information encapsulated by the peak (rather than the mean) opponent motion energy and the predominantly local nature of imbalances in motion energy that can arise in contrast-modulated static noise, it is possible to readily model the patterns of empirical results found.

The Ferrier Lecture 2004 what can transcranial magnetic stimulation tell us about how the brain works?

Transcranial magnetic stimulation (TMS) is a technique whereby parts of the cerebral cortex and underlying white matter can be excited by a brief electrical current induced by a similarly brief, rapidly fluctuating magnetic field which is itself produced by rapidly discharging a current through an insulated coil held against the scalp. When combined with magnetic resonance structural and functional images of the subject's brain, the stimulation can be directed at specific cortical areas. Over a period of only 15 years, TMS has revealed hitherto unsuspected aspects of brain function, such as the role of distant parts of the brain in recovery from stroke, and has helped to resolve several previously intractable disputes, such as the neuronal basis of conscious awareness. This article describes and discusses the origins and nature of TMS, its applications and limitations, and its especial usefulness in conjunction with other techniques of evaluating or imaging brain activity.

The direction-selective contrast response of area 18 neurons is different for first- and second-order motion

Cortical neurons selective for the direction of motion often exhibit some limited response to motion in their nonpreferred directions. Here we examine the dependence of neuronal direction selectivity on stimulus contrast, both for first-order (luminance-modulated, sine-wave grating) and second-order (contrast-modulated envelope) stimuli. We measured responses from single neurons in area 18 of cat visual cortex to both kinds of moving stimuli over a wide range of contrasts (1.25-80%). Direction-selective contrast response functions (CRFs) were calculated as the preferred-minus-null difference in average firing frequency as a function of contrast. We also applied receiver operating characteristic analysis to our CRF data to obtain neurometric functions characterizing the potential ability of each neuron to discriminate motion direction at each contrast level tested. CRFs for sine-wave gratings were usually monotonic; however, a substantial minority of neurons (35%) exhibited nonmonotonic CRFs (such that the degree of direction selectivity decreased with increasing contrast). The underlying preferred and nonpreferred direction CRFs were diverse, often having different shapes in a given neuron. Neurometric functions for direction discrimination showed a similar degree of heterogeneity, including instances of nonmonotonicity. For contrast-modulated stimuli, however, CRFs for either carrier or envelope contrast were always monotonic. In a given neuron, neurometric thresholds were typically much higher for second- than for first-order stimuli. These results demonstrate that the degree of a cell's direction selectivity depends on the contrast at which it is measured, and therefore is not a characteristic parameter of a neuron. In general, contrast response functions for first-order stimuli were very heterogeneous in shape and sensitivity, while those for second-order stimuli showed less sensitivity and were quite stereotyped in shape.

Human cortical responses to contrast modulations of visual noise

We studied visual evoked potentials (VEPs) elicited by second-order contrast modulations of binary dynamic noise and first-order luminance modulations. Using a 3-point Laplacian operator centred on Oz, we found that contrast modulations of both low and higher spatial frequencies elicited a negative component whose latency was about 200 ms. The latency of this component was significantly longer than that of the early Laplacian components to first-order luminance modulations. These findings could be due to slower first-stage linear filters and additional processing stages of the second-order pathway. The topographical analysis of scalp recorded VEPs to central and half-field stimulation has suggested that the responses to second-order patterns are likely to be generated by neuronal structures within the primary visual cortex which may have inputs from extrastriate neurons via feedback connections.

A common mechanism for the perception of first-order and second-order apparent motion

A common mechanism for perceiving first-order, luminance-defined, and second-order, texture-contrast defined apparent motion between two element locations is indicated by: (1) transitivity--whether or not motion is perceived is inter-changeably affected by activationally equivalent luminance and contrast changes at each location, (2) local integration--whether or not motion is perceived depends on the net activation change resulting from simultaneous background-relative luminance and background-relative contrast changes at the same element location, and (3) inseparability--apparent motion is not perceived through independent first- or second-order mechanisms when luminance and contrast co-vary at the same location. These results, which are predicted by the response characteristics of directionally selective cells in areas V1, MT, and MST, are not instead attributable to changes in the location of the most salient element (third-order motion), attentive feature tracking, or artifactual first-order motion. Their inconsistency with Lu and Sperling's [Lu, Z., Sperling, G. (1995a). Attention-generated apparent motion. Nature 377, 237, Lu, Z., Sperling, G. (2001). Three-systems theory of human visual motion perception: review and update. Journal of the Optical Society of America A 18, 2331] model, which specifies independent first- and second-order mechanisms, may be due to computational requirements particular to the motion of discrete objects with distinct boundaries defined by spatial differences in luminance, texture contrast, or both.

From 1D to 2D via 3D: dynamics of surface motion segmentation for ocular tracking in primates

In primates, tracking eye movements help vision by stabilising onto the retinas the images of a moving object of interest. This sensorimotor transformation involves several stages of motion processing, from the local measurement of one-dimensional luminance changes up to the integration of first and higher-order local motion cues into a global two-dimensional motion immune to antagonistic motions arising from the surrounding. The dynamics of this surface motion segmentation is reflected into the various components of the tracking responses and its underlying neural mechanisms can be correlated with behaviour at both single-cell and population levels. I review a series of behavioural studies which demonstrate that the neural representation driving eye movements evolves over time from a fast vector average of the outputs of linear and non-linear spatio-temporal filtering to a progressive and slower accurate solution for global motion. Because of the sensitivity of earliest ocular following to binocular disparity, antagonistic visual motion from surfaces located at different depths are filtered out. Thus, global motion integration is restricted within the depth plane of the object to be tracked. Similar dynamics were found at the level of monkey extra-striate areas MT and MST and I suggest that several parallel pathways along the motion stream are involved albeit with different latencies to build-up this accurate surface motion representation. After 200-300 ms, most of the computational problems of early motion processing (aperture problem, motion integration, motion segmentation) are solved and the eye velocity matches the global object velocity to maintain a clear and steady retinal image.

Deficits in the processing of local and global motion in very low birthweight children

This study evaluated the impact of premature birth on the development of local and global motion processing in a group of very low birthweight (<1500 g), 5- to 8-year-old children. Sensitivity to first- and second-order local motion stimuli and coherence thresholds for global motion in random dot kinematograms were measured. Relative to full-term controls, premature children showed deficits on all three aspects of motion processing. These problems could not be accounted for by stereo deficits, amblyopia, or attentional problems. A history of mild retinopathy of prematurity and/or intraventricular hemorrhage increased risk, but deficits were observed in some children with no apparent ocular or cerebral pathology. It is important to note that, despite the observed group differences, individual profiles of performance did vary; the results suggest that these three forms of motion processing may involve separate neural mechanisms. These findings serve to increase our understanding of the organization and functional development of motion-processing subsystems in humans, and of the impact of prematurity and associated complications on visual development.

Putting order into the development of sensitivity to global motion

We studied differences in the development of sensitivity to first-versus second-order global motion by comparing the motion coherence thresholds of 5-year-olds and adults tested at three speeds (1.5, 6, and 9 degrees s(-1)). We used Random Gabor Kinematograms (RGKs) formed with luminance-modulated (first-order) or contrast-modulated (second-order) concentric Gabor patterns with a sinusoidal spatial frequency of 3c deg(-1). To achieve equal visibility, modulation depth was set at 30% for first-order Gabors and at 100%, for second-order Gabors. Subjects were 24 adults and 24 5-year-olds. For both first- and second-order global motion, the motion coherence threshold of 5-year-olds was less mature for the slowest speed (1.5 degrees s(-1)) than for the two faster speeds (6 and 9 degrees s(-1)). In addition, at the slowest speed, the immaturity was greater for second-order than for first-order global motion. The findings suggest that the extrastriate mechanisms underlying the perception of global motion are different, at least in part, for first- versus second-order signals and for slower versus faster speeds. They also suggest that those separate mechanisms mature at different rates during middle childhood.

Contrast-reversing global-motion stimuli reveal local interactions between first- and second-order motion signals

Motion perception appears to be mediated by, at least, two systems: a first-order and a second-order system. To investigate the degree of interaction between these systems, we used a contrast-reversing global-motion stimulus in which the signal dots reverse their contrast polarity as they move. In response to such a stimulus, fullwave-rectifying second-order units would signal motion in the displacement direction and first-order units would signal motion in the opposite direction (reverse-phi motion). If these signals were of equal strength, then any inhibitory interaction between them would lead to motion nulling. Such a situation would account for the failure to perceive coherent motion with such a stimulus in a previous study [Vis. Res. 34 (1994) 2849]. In order to test for this possibility we manipulated the stimulus in order to reduce the strength of the second-order response relative to the first-order response. This was achieved by: decreasing dot contrast; increasing stimulus eccentricity; and increasing dot speed. These manipulations resulted in an increase in the perception of (first-order mediated) reverse-phi motion. We conclude that interaction between first- and second-order motion signals occur at the local-motion-pooling level.

Investigating local network interactions underlying first- and second-order processing

We compared the spatial lateral interactions for first-order cues to those for second-order cues, and investigated spatial interactions between these two types of cues. We measured the apparent modulation depth of a target Gabor at fixation, in the presence and the absence of horizontally flanking Gabors. The Gabors' gratings were either added to (first-order) or multiplied with (second-order) binary 2-D noise. Apparent "contrast" or modulation depth (i.e., the perceived difference between the high and low luminance regions for the first-order stimulus, or between the high and low contrast regions for the second-order stimulus) was measured with a modulation depth-matching paradigm. For each observer, the first- and second-order Gabors were equated for apparent modulation depth without the flankers. Our results indicate that at the smallest inter-element spacing, the perceived reduction in modulation depth is significantly smaller for the second-order than for the first-order stimuli. Further, lateral interactions operate over shorter distances and the spatial frequency and orientation tuning of the suppression effect are broader for second- than first-order stimuli. Finally, first- and second-order information interact in an asymmetrical fashion; second-order flankers do not reduce the apparent modulation depth of the first-order target, whilst first-order flankers reduce the apparent modulation depth of the second-order target.

Contour integration and cortical processing

Our understanding of visual processing in general, and contour integration in particular, has undergone great change over the last 10 years. There is now an accumulation of psychophysical and neurophysiological evidence that the outputs of cells with conjoint orientation preference and spatial position are integrated in the process of explication of rudimentary contours. Recent neuroanatomical and neurophysiological results suggest that this process takes place at the cortical level V1. The code for contour integration may be a temporal one in that it may only manifest itself in the latter part of the spike train as a result of feedback and lateral interactions. Here we review some of the properties of contour integration from a psychophysical perspective and we speculate on their underlying neurophysiological substrate.

First-order and second-order motion: neurological evidence for neuroanatomically distinct systems

An unresolved issue in visual motion perception is how distinct are the processes underlying 'first-order' and 'second-order' motion. The former is defined by spatio-temporal variations of luminance and the latter by spatio-temporal variations in other image attributes such as contrast or depth, for example. Using neuroimaging and psychophysics we present data from four neurological patients with unilateral and mostly cortical infarcts, which strongly suggest that first- and second-order motion have a different neural substrate. We found that from the early stages of processing, these two types of motions are mediated by two distinct pathways: first-order motion is carried out by mechanisms along the dorsal pathway in the occipital lobe, while the second-order motion by mechanisms mostly along the ventral pathway. The data reported here also suggest that different cortical regions may be in charge of processing direction-discrimination in second-order motion defined by different second-order attributes.

How is complex second-order motion processed?

Converging psychophysical and electrophysiological evidence suggests that first-order (luminance-defined) complex motion types i.e., radial and rotational motion, are processed by specialized extrastriate motion mechanisms. We ask whether radial and rotational second-order (texture-defined) motion patterns are processed in a similar manner. The motion sensitivity to translating, radiating and rotating motion patterns of both first-order (luminance-modulated noise) and second-order (contrast-modulated noise) were measured for patterns presented at four different exposure durations (106, 240, 500 and 750 ms). No significant difference in motion sensitivity was found across motion type for the first-order motion class across exposure duration (i.e., from 240 to 750 ms) whereas direction-identification thresholds for radiating and rotating second-order motion were significantly greater than that of the second-order translational stimuli. Furthermore, thresholds to all second-order motion stimuli increased at a significantly faster rate with decreasing exposure duration compared to those of first-order motion. Interestingly, simple and complex second-order thresholds increased at similar rates. Taken together, the results suggest that complex second-order motion is not analyzed in a sequential manner. Rather, it seems that the same 'hard-wired' mechanisms responsible for complex first-order motion processing also mediate complex second-order motion, but not before the pre-processing (i.e., rectification) of local second-order motion signals.

Attentional modulation of threshold sensitivity to first-order motion and second-order motion patterns

Previous studies [e.g. Vision Research 40 (2000) 173] have shown that when observers are required to selectively attend to one of two, spatially-adjacent patches containing either first-order (luminance-defined) or second-order (contrast-defined) motion, threshold sensitivity for identifying the direction of second-order motion, but not first-order motion, is enhanced for the attended stimuli. The processing of second-order motion, unlike first-order motion, may, therefore, require attention. However, other studies have found little evidence for differential effects of attention on the processing of first-order and second-order motion [Investigative Ophthalmology and Visual Science 42(4) (2001) 5061]. We investigated the effects of attention instructions on the ability of observers to identify the directions and spatial orientations of luminance-defined and contrast-defined motion stimuli. Pairs of motion stimuli were presented simultaneously and threshold performance was measured over a wide range of drift temporal frequencies and stimulus durations. We found: (1) direction discrimination thresholds for attended motion stimuli were lower than those for unattended stimuli for both types of motion. The magnitude of this effect was reduced when the observers were not given prior knowledge of which patch of motion (attended or unattended) they had to judge first. (2) Direction discrimination for first-order motion was similarly affected at all temporal frequencies and durations examined, but for second-order motion the effects of attention depended critically on the drift temporal frequency and stimulus duration used. (3) Orientation discrimination showed little or no influence of attention instructions. Thus, whether or not attention influences the processing of second-order motion depends crucially on the precise stimulus parameters tested. Furthermore under appropriate conditions the processing of first-order motion is also influenced by attention, albeit to a lesser extent than second-order motion.

Normal and anomalous development of visual motion processing: motion coherence and 'dorsal-stream vulnerability'

Directional motion processing is a pervasive and functionally important feature of the visual system. Behavioural and VEP studies indicate that it appears as a cortical function after about 7 weeks of age, with global processing, motion based segmentation, and the use of motion in complex perceptual tasks emerging shortly afterwards. A distinct, subcortical motion system controls optokinetic nystagmus (OKN) from birth, showing characteristic monocular asymmetries which disappear as binocular cortical function takes over in normal development. Asymmetries in cortical responses are linked to this interaction in a way that is not yet fully understood. Beyond infancy, a range of developmental disorders show a deficit of global motion compared to global form processing which we argue reflects a general 'dorsal-stream vulnerability'.

Neuroimaging of direction-selective mechanisms for second-order motion

Psychophysical findings have revealed a functional segregation of processing for 1st-order motion (movement of luminance modulation) and 2nd-order motion (e.g., movement of contrast modulation). However neural correlates of this psychophysical distinction remain controversial. To test for a corresponding anatomical segregation, we conducted a new functional magnetic resonance imaging (fMRI) study to localize direction-selective cortical mechanisms for 1st- and 2nd-order motion stimuli, by measuring direction-contingent response changes induced by motion adaptation, with deliberate control of attention. The 2nd-order motion stimulus generated direction-selective adaptation in a wide range of visual cortical areas, including areas V1, V2, V3, VP, V3A, V4v, and MT+. Moreover, the pattern of activity was similar to that obtained with 1st-order motion stimuli. Contrary to expectations from psychophysics, these results suggest that in the human visual cortex, the direction of 2nd-order motion is represented as early as V1. In addition, we found no obvious anatomical segregation in the neural substrates for 1st- and 2nd-order motion processing that can be resolved using standard fMRI.

Physiological evidence of interaction of first- and second-order motion processes in the human visual system: a magnetoencephalographic study

Humans have several mechanisms for the visual perception of motion, including one that is luminance-based (first-order) and another that is luminance-independent (second-order). Recent psychophysical studies have suggested that significant interaction occurs between these two neural processes. We investigated whether such interactions are represented as neural activity measured by magnetoencephalography (MEG). The second-order motion of a drifting sinusoidal grating, which is defined by the speed of the dot motion, did not generate a response. Apparent motion (AM) of the square area, defined by the speed of randomly moving dots, evoked a magnetic response whose latency and amplitude changed with the distance that the area moved (a second-order characteristic), though the response properties were significantly different from those for the first-order AM. AM, defined by both first- and second-order attributes, evoked an MEG response and the latencies and the amplitudes were distributed between those for the first- and second-order motions. The cortical source of the response was estimated to be around MT+. The results show a distinct difference in the neural processing of the second-order motion that cannot be explained by the difference in visibility, and they indicate that the interaction of the neural processes underlying first- and second-order motion detection occurs before the MEG response. Our study provides the first physiological evidence of a neural interaction between the two types of early motion detection.

The role of local grouping and global orientation contrast in perception of orientation-modulated textures

We explored the contribution to perception of orientation-modulated textures of visual processes selective either for orientation contrast or orientation grouping. To distinguish between these two processes we manipulated the axis of local grouping of texture elements independently of the direction of global orientation modulation. The general question posed was whether visibility of texture structure (measured as threshold for discriminating spatial-frequency of texture structure) is dependent on the magnitude of orientation contrast, strength and direction of local grouping, or some combination of the two. We demonstrated that the factor of primary importance is the amplitude of global orientation contrast rather than the presence of local grouping content. Using orientation-interleaved textures (containing two superimposed textures modulated around orthogonal orientations), we further showed that orientation single-opponent processes are a more likely candidate for detecting orientation contrast than double-opponent processes.

Temporal properties of the visual responses to luminance and contrast modulated noise

Vision is sensitive to first-order luminance modulations and second-order modulations of carrier contrast. Our knowledge of the temporal properties of second-order vision is insufficient and contradictory. Using temporal summation and reaction time paradigms, we found that the type of visual noise (static or dynamic) determines the temporal properties of the responses to luminance and contrast modulations. In the presence of static noise, the temporal responses to both types of modulation of low and higher spatial frequencies were transient. When dynamic noise was used, the temporal responses to luminance and contrast modulations of higher spatial frequencies were sustained. At low spatial frequency, however, luminance modulations elicited transient responses, while contrast modulated dynamic noise produced sustained responses. The reaction times to near-threshold contrast modulations of low spatial frequency were slower than those to first-order patterns and they did not significantly differ at modulations of higher spatial frequency. The results suggest that the temporal characteristics of first-stage linear filters which feed the second-order pathway may determine the temporal responses to contrast modulated noise.

Cerebral regions processing first- and higher-order motion in an opposed-direction discrimination task

Using PET, we studied the processing of different types of motion in an opposed-direction discrimination task. We used first-order motion and two types of higher-order motion (presented as moving gratings with stripes defined by flickering texture and kinetic boundaries, respectively). In these experiments, we found that all types of motion activate a common set of cortical regions when comparing a direction discrimination task to a detection of the dimming of the fixation point. This set includes left hV3A, bilateral hMT/V5+ and regions in the middle occipital gyrus, bilateral activations in the posterior and anterior parts of the intraparietal sulcus, bilateral precentral gyrus, medial frontal cortex and regions in the cerebellum. No significant differences were observed between different types of motion, even at low statistical thresholds. From this we conclude that, under our experimental conditions, the same cerebral regions are involved in the processing of first-order and higher-order motion in an opposed-direction discrimination task.

Longer VEP latencies and slower reaction times to the onset of second-order motion than to the onset of first-order motion

We compared visual evoked potentials and psychophysical reaction times to the onset of first- and second-order motion. The stimuli consisted of luminance-modulated (first-order) and contrast-modulated (second-order) 1 cpd vertical sine-wave gratings drifting rightward for 140 ms at a velocity of 6 degrees /s. For each condition, we analysed the latencies and peak-to-baseline amplitudes of the P1 and N2 peaks recorded at Oz. For first-order motion, both P1 and N2 peaks were present at low (3%) contrast (i.e., depth modulations) whereas for second-order motion they appeared only at higher (25%) contrasts. When the two types of motion were equated for visibility, responses were slower for second-order motion than for first-order motion: about 44 ms slower for P1 latencies, 53 ms slower for N2 latencies, and 76 ms slower for reaction times. The longer VEP latencies for second-order motion support models that postulate additional processing steps for the extraction of second-order motion. The slower reaction time to the onset of second-order motion suggests that the longer neurophysiological analysis translates into slower detection.

Failure of direction identification for briefly presented second-order motion stimuli: evidence for weak direction selectivity of the mechanisms encoding motion

We sought to investigate why the direction of second-order motion, unlike first-order motion, cannot be identified when the stimulus exposure duration is brief (<200 ms). In a series of experiments observers identified both the orientation (vertical or horizontal) and the direction (left, right, down or up) of a drifting sinusoidal modulation (0.93 c/ degrees ) in either the luminance (first order) or the contrast (second order) of a two-dimensional noise carrier. All motion stimuli were equated for visibility, and the duration was varied using the method of constant stimuli. Performance was measured for second-order motion over a range of drift temporal frequencies (0.63-5.04 Hz) and for first-order motion stimuli composed of two, opposite drifting modulations in luminance of unequal modulation depth. Orientation-identification performance was nearly 100% correct for both first-order and second-order motion stimuli, even at the briefest stimulus duration tested (26.49 ms). Direction identification for first-order motion was also typically good with brief presentations, but was poor for second-order motion when the exposure duration was < approximately 200 ms. Importantly increasing either the drift temporal frequency of second-order motion or the bidirectional nature of the first-order motion patterns produced comparable levels of performance for the two varieties of motion (i.e. the minimum duration required for reliable direction identification could be equated). As orientation-identification performance for the first-order and second-order motion stimuli was comparably good and minimally affected by duration, the marked differences on the direction-identification task must be specific to mechanisms that encode drift direction, rather than spatial structure. We propose that second-order motion detectors are much less selective for stimulus direction than first-order motion sensors, and thus are more susceptible to the deleterious effects of limiting stimulus duration (which introduces spurious motion in the opposite direction, particularly at low drift rates). Alternative explanations based on the delayed propagation of second-order motion signals or the temporal characteristics of the underlying motion mechanisms are not supported by our findings.

Isolation of stimulus characteristics contributing to Weber's law for position

To examine the independent contribution of various stimulus characteristics to positional judgements, we measured vernier alignment performance for three types of Gabor stimuli. In one, only the contrast envelope of the upper and lower stimulus elements was offset, with the luminance-modulated carrier grating remaining in alignment. In the second, only the carrier grating was offset. In the third, both carrier and envelope were offset together. Performance was examined over a range of element separations. When both cues are available, thresholds for small separations are dominated by carrier offset information and are inversely proportional to carrier frequency. At large separations, thresholds are governed by the spatial scale characteristics of the envelope. For broad-band stimuli such as lines, bars or dots typically used for vernier acuity, their higher frequency content can be used when separations are small, but as separation increases a smooth transition between the scales that determine threshold results in the continuum known as Weber's law for position. That is, with increasing separation, larger scales must be used, and thresholds increase in direct proportion to 1/frequency.

The duration of 3-d form analysis in transformational apparent motion

Transformational apparent motion (TAM) occurs when a figure changes discretely from one configuration to another overlapping configuration. Rather than an abrupt shape change, the initial shape is perceived to transform smoothly into the final shape as if animated by a series of intermediate shapes. We find that TAM follows an analysis of form that takes 80-140 msec. Form analysis can function both at and away from equiluminance and can occur over contours defined by uniform regions as well as outlines. Moreover, the forms analyzed can be 3-D, resulting in motion paths that appear to smoothly project out from or into the stimulus plane. The perceived transformation is generally the one that involves the least change in the shape or location of the initial figure in a 3-D sense. We conclude that perception of TAM follows an analysis of 3-D form that takes approximately 100 msec. This stage of form analysis may be common to both TAM and second-order motion.

Higher-order motion processing in the pulvinar

Thalamic nuclei have long been considered as passive relay stations for sensory signals en route to the cerebral cortex, where higher level processing occurs. In recent years, it has been proposed that thalamic nuclei may actively participate in the processing of specific information in conjunction with cortical areas. In support of this hypothesis, we recently discovered that neurons in the main extrageniculate visual nucleus, the pulvinar, exhibit higher-order visual properties that were, until now, only associated with higher-order cortical areas. Pulvinar neurons can indeed code the veridical direction of a moving plaid pattern, indicating that these cells can integrate ambiguous signals into a coherent percept. This finding as well as our demonstration that there are cortico-thalamo-cortical loops involved in complex motion analysis open promising avenues in unraveling the function of the pulvinar complex in normal vision.

Processing of second-order stimuli in the visual cortex

Naturally occurring visual stimuli are rich in examples of objects delineated from their backgrounds simply by differences in luminance, so-called first-order stimuli, as well as those defined by differences of contrast or texture, referred to as second-order stimuli. Here we provide a brief overview of visual cortical processing of second-order stimuli, as well as some comparative background on first-order processing, concentrating on single-unit neurophysiology, but also discussing relationships to human psychophysics and to neuroimaging. The selectivity of visual cortical neurons to orientation, spatial frequency, and direction of movement of first-order, luminance-defined stimuli is conventionally understood in terms of simple linear filter models, albeit with some minor nonlinearities such as thresholding and gain control. However, these kinds of models fail entirely to account for responses of neurons to second-order stimuli such as contrast envelopes, illusory contours, or texture borders. Second-order stimuli constructed from sinusoidal components have been used to analyze the neurophysiological mechanisms of such responses; these experiments demonstrate that the same neuron can exhibit three distinct kinds of tuning to spatial frequency, and also to orientation. These results can be understood in terms of a type of nonlinear 'filter-->rectify-->filter' model, which has been widely used in human psychophysics. Finally, several general issues will be discussed, including potential artifacts in experiments with second-order stimuli, and strategies for avoiding or controlling for them; caveats about definitions of first- vs. second-order mechanisms and stimuli; the concept of form-cue invariance; and the functional significance of second-order processing.

Interaction of first- and second-order direction in motion-defined motion

Motion-defined motion can play a special role in the discussion of whether one or two separate systems are required to process first- and second-order information because, in contrast to other second-order stimuli, such as contrast-modulated contours, motion detection cannot be explained by a simple input nonlinearity but requires preprocessing by motion detectors. Furthermore, the perceptual quality that defines an object (motion on the object surface) is identical to that which is attributed to the object as an emergent feature (motion of the object), raising the question of how these two object properties are linked. The interaction of first- and second-order information in such stimuli has been analyzed previously in a direction-discrimination task, revealing some cooperativity. Because any comprehensive integration of these two types of motion information should be reflected in the most fundamental property of a moving object, i.e., the direction in which it moves, we now investigate how motion direction is estimated in motion-defined objects. Observers had to report the direction of moving objects that were defined by luminance contrast or in random-dot kinematograms by differences in the spatiotemporal properties between the object region and the random-noise background. When the dots were moving coherently with the object (Fourier motion), direction sensitivity resembled that for luminance-defined objects, but performance deteriorated when the dots in the object region were static (drift-balanced motion). When the dots on the object surface were moving diagonally relative to the object direction (theta motion), the general level of accuracy declined further, and the perceived direction was intermediate between the veridical object motion direction and the direction of dot motion, indicating that the first- and second-order velocity vectors are somehow pooled. The inability to separate first- and second-order directional information suggests that the two corresponding subsystems of motion processing are not producing independent percepts and provides clues for possible implementations of the two-layer motion-processing network.

Centrifugal bias for second-order but not first-order motion

Limited-lifetime Gabor stimuli were used to assess both first- and second-order motion in peripheral vision. Both first- and second-order motion mechanisms were present at a 20-deg eccentricity. Second-order motion, unlike first-order, exhibits a bias for centrifugal motion, suggesting a role for the second-order mechanism in optic flow processing.

Three-systems theory of human visual motion perception: review and update

Lu and Sperling [Vision Res. 35, 2697 (1995)] proposed that human visual motion perception is served by three separate motion systems: a first-order system that responds to moving luminance patterns, a second-order system that responds to moving modulations of feature types-stimuli in which the expected luminance is the same everywhere but an area of higher contrast or of flicker moves, and a third-order system that computes the motion of marked locations in a "salience map," that is, a neural representation of visual space in which the locations of important visual features ("figure") are marked and "ground" is unmarked. Subsequently, there have been some strongly confirmatory reports: different gain-control mechanisms for first- and second-order motion, selective impairment of first- versus second- and/or third-order motion by different brain injuries, and the classification of new third-order motions, e.g., isoluminant chromatic motion. Various procedures have successfully discriminated between second- and third-order motion (when first-order motion is excluded): dual tasks, second-order reversed phi, motion competition, and selective adaptation. Meanwhile, eight apparent contradictions to the three-systems theory have been proposed. A review and reanalysis here of the new evidence, pro and con, resolves the challenges and yields a more clearly defined and significantly strengthened theory.

Hierarchy of spatial interactions in the processing of contrast-defined contours

Both psychophysical and neurophysiological evidence suggest that there are two visual cortical processing streams, a linear stream that processes first-order stimuli and a nonlinear stream that also processes second-order stimuli. This evidence also suggests that before the extraction of the second-order signal, the nonlinear pathway broadly but not completely pools signals across initial linear filters that encode the orientation of the carrier of the second-order signal. The evidence suggests that such pooling does not occur across carrier spatial frequencies. We show that similar results are obtained with repulsion tilt illusions but not with attraction effects. Attraction effects exhibit complete orientation crossover (while retaining spatial frequency selectivity), perhaps indicating higher-level processing; an experiment on interocular transfer of the effects supported this conclusion.

Detection of constrast-defined shape

We assessed the accuracy of contrast-defined shape detection of stimuli of constant aspect ratio, namely, circular bandpass stimuli whose radii were sinusoidally varied about a mean radius. Performance for these contrast-defined shapes, which we show is determined by the global rather than the local attributes of the stimulus, is 2-8 times worse than that for their luminance-defined counterparts, suggesting separate processing limitations. By spatially and orientationally filtering the two-dimensional fractal-noise carriers of which these stimuli were composed, we determined whether there are specific rules concerning the spatial and orientational input to shape detectors from mechanisms sensitive to the carrier structure. The results suggest that second-order circularity detectors receive mixed input from spatial-frequency-tuned and orientationally tuned cells.

Interaction between first- and second-order orientation channels revealed by the tilt illusion: psychophysics and computational modelling

This paper examines the interaction between first- and second-order contours in the orientation domain. Using the simultaneous tilt illusion (TI), we show that the apparent rotation of a vertical test grating away from that of a surrounding inducing grating (repulsion effect) occurs when both the inducing and test grating are either first- or second-order. Furthermore, a significant repulsion effect is obtained when a first-order inducing grating surrounds a second-order test. If lateral inhibitory interactions between populations of orientation selective neurons provides a plausible explanation for orientation repulsion effects [Blakemore, C. B. Carpenter, R. H. S. & Georgeson, M. A. (1970) Nature, 228, 37-39], it is likely that the cue-invariant mechanisms that encodes the orientation of first- and second-order contours also exhibit inhibitory interactions. A two-channel computational model of orientation encoding is presented where one channel encodes only first-order stimuli while the second channel encodes both first- and second-order contours. In addition to predicting the orientation repulsion effects we observed, the model also provides a functional account of orientation attraction effects in terms of the responses of populations of orientation-tuned neurons.

No local cancellation between directionally opposed first-order and second-order motion signals

Despite strong converging evidence that there are separate mechanisms for the processing of first-order and second-order motion, the issue remains controversial. Qian, Andersen and Adelson (J. Neurosci., 14 (1994), 7357-7366) have shown that first-order motion signals cancel if locally balanced. Here we show that this is also the case for second-order motion signals, but not for a mixture of first-order and second-order motion even when the visibility of the two types of stimulus is equated. Our motion sequence consisted of a dynamic binary noise carrier divided into horizontal strips of equal height, each of which was spatially modulated in either contrast or luminance by a 1.0 c/deg sinusoid. The modulation moved leftward or rightward (3.75 Hz) in alternate strips. The single-interval task was to identify the direction of motion of the central strip. Three conditions were tested: all second-order strips, all first-order strips, and spatially alternated first-order and second-order strips. In the first condition, a threshold strip height for the second-order strips was obtained at a contrast modulation depth of 100%. In the second condition, this height was used for the first-order strips, and a threshold was obtained in terms of luminance contrast. These two previously-obtained threshold values were used to equate visibility of the first-order and second-order components in the third condition. Direction identification, instead of being at threshold, was near-perfect for all observers. We argue that the first two conditions demonstrate local cancellation of motion signals, whereas in the third condition this does not occur. We attribute this non-cancellation to separate processing of first-order and second-order motion inputs.

A motion aftereffect seen more strongly by the non-adapted eye: evidence of multistage adaptation in visual motion processing

We found that the motion aftereffect measured using a directionally ambiguous counterphase grating (flicker MAE) can be stronger when it is measured for the non-adapted eye than when measured for the adapted eye. The monocularly viewed adaptation stimulus was the movement of a missing-fundamental grating (2f+3f motion), for which the movement of the higher-order spatial structure was dominantly perceived, while the first-order structure was physically moving in the opposite direction. For observers who perceived the MAE consistently in the direction opposite to the movement of the higher-order structures, the MAE was larger for the non-adapted eye than for the adapted eye. This finding of 'over-100% transfer' invalidates the standard view that the IOT is a direct measure of the binocularity of the adapted neurones. In addition, the finding provides convincing support for the hypothesis that the flicker MAE reflects adaptation at multiple processing stages

A transient deficit of motion perception in human

We studied the motion perception abilities in a young adult, SF, who had her right occipito-temporal cortices resected to treat epilepsy. Following resection, SF showed transient deficits of both first- and second-order motion perception that recovered to normal within weeks. Previous human studies have shown either first- or second n order motion deficits that have lasted months or years after cerebral damage. SF also showed a transient defect in processing of shape-from-motion with normal perception of shape from non-motion cues. Furthermore, she showed greatly increased reaction times for a mental rotation task, but not for a lexical decision task. The nature and quick recovery of the deficits in SF resembles the transient motion perception deficit observed in monkey following ibotenic acid lesions, and provides additional evidence that humans possess specialized cortical areas subserving similar motion perception functions.

Impoverished second-order input to global linking in human vision

Recent evidence points to the importance of global operations across spatial regions larger than individual cortical receptive fields. Studies of contour integration and motion trajectory detection suggest that network operations between local detectors underlie the encoding of extended contours in space and extended trajectories in motion. Here we ask whether such network operations also occur between second-order-detectors known to exist in visual cortex. We compared performance for stimuli composed of either first-order or second-order elements equated for visibility, and we show that unlike the first-order case, there is little or no linking interaction between local second-order detectors. Near chance performance was found for elements defined by second-order attributes when observers had to identify either an elongated spatial contour or an extended motion trajectory embedded in noise elements. This implies that the network operations thought to underlie these two global tasks receive, at best, an impoverished input from local detectors that encode second-order image attributes.

Orientation integration in detection and discrimination of contrast-modulated patterns

Orientation detection and discrimination thresholds were measured for Gabor 'envelopes' formed from contrast-modulation of luminance 'carriers'. Consistent with previous research differences between carrier and envelope orientation had no effect on sensitivity to envelopes. Using plaid carriers in which the proportion of contrast modulation 'carried' by each plaid component was systematically manipulated, it was shown that this tolerance to carrier-envelope orientation difference reflects linear summation across orientation indicative of a single second-stage channel coding for contrast-defined structure. That contrast envelopes did not exhibit linear summation across spatial-frequency, nor across combinations of orientation and spatial-frequency differences, suggests that these second-order channels operate only within certain spatial scales. Using arrays of Gabor micropatterns as carriers in which the orientation distribution of the carriers was manipulated independently of the difference between envelope orientation and mean carrier orientation, it was further demonstrated that the locus of orientation integration must occur prior to envelope detection. In the context of two-stage models that incorporate a non-linearity between the stages, the pattern of results obtained is consistent with the operation of an orientation pooling process between first-stage and second-stage channels, analogous to having all filters of the first-stage feed into all filters of the second-stage within the same spatial-frequency band.

Orientation processing mechanisms revealed by the plaid tilt illusion

The tilt after-effect (TAE) and tilt illusion (TI) have revealed a great deal about the nature of orientation coding of 1-dimensional (1D) lines and gratings. Comparatively little research however has addressed the mechanisms responsible for encoding the orientation of 2-dimensional (2D) plaid stimuli. A multi-stage model of edge detection has recently been proposed [Georgeson, M. A. (1998) Image & Vision Computing, 16(6-7), 389-405] to account for the perceived structure of a plaid stimulus that incorporates extraction of the zero-crossings (ZCs) of the plaid. Data is presented showing that the ZCs of a plaid inducing stimulus can interact with vertical grating test stimulus to induce a standard tilt illusion. However, by considering the second-order structure of a plaid rather than ZCs, it was shown that the perceived orientation of the vertical test grating results from the combination of orientation illusions due to the first- and second-order components of an inducing plaid. The data suggest that the mechanisms encoding the orientation of second-order contours are similar to, and interact directly with, those that encode first-order contours.

Perception of Fourier and non-Fourier motion by larval zebrafish

A moving grating elicits innate optomotor behavior in zebrafish larvae; they swim in the direction of perceived motion. We took advantage of this behavior, using computer-animated displays, to determine what attributes of motion are extracted by the fish visual system. As in humans, first-order (luminance-defined or Fourier) signals dominated motion perception in fish; edges or other features had little or no effect when presented with these signals. Humans can see complex movements that lack first-order cues, an ability that is usually ascribed to higher-level processing in the visual cortex. Here we show that second-order (non-Fourier) motion displays induced optomotor behavior in zebrafish larvae, which do not have a cortex. We suggest that second-order motion is extracted early in the lower vertebrate visual pathway.

First- and second-order processing in transient stereopsis

Large-field stimuli were used to investigate the interaction of first- and second-order pathways in transient-stereo processing. Stimuli consisted of sinewave modulations in either the mean luminance (first-order stimulus) or the contrast (second-order stimulus) of a dynamic-random-dot field. The main results of the present study are that: (1) Depth could be extracted with both the first-order and second-order stimuli; (2) Depth could be extracted from dichoptically mixed first- and second-order stimuli, however, the same stimuli, when presented as a motion sequence, did not result in a motion percept. Based upon these findings we conclude that the transient-stereo system processes both first- and second-order signals, and that these two signals are pooled prior to the extraction of transient depth. This finding of interaction between first- and second-order stereoscopic processing is different from the independence that has been found with the motion system.