Responses of complex cells in area 17 of the cat to bi-vectorial transparent motion

Visual motion integration of bidirectional transparent motion in mouse opto-locomotor reflexes

Visual motion perception depends on readout of direction selective sensors. We investigated in mice whether the response to bidirectional transparent motion, activating oppositely tuned sensors, reflects integration (averaging) or winner-take-all (mutual inhibition) mechanisms. We measured whole body opto-locomotor reflexes (OLRs) to bidirectional oppositely moving random dot patterns (leftward and rightward) and compared the response to predictions based on responses to unidirectional motion (leftward or rightward). In addition, responses were compared to stimulation with stationary patterns. When comparing OLRs to bidirectional and unidirectional conditions, we found that the OLR to bidirectional motion best fits an averaging model. These results reflect integration mechanisms in neural responses to contradicting sensory evidence as has been documented for other sensory and motor domains.

Unconscious Biases in Neural Populations Coding Multiple Stimuli

Throughout the nervous system, information is commonly coded in activity distributed over populations of neurons. In idealized situations where a single, continuous stimulus is encoded in a homogeneous population code, the value of the encoded stimulus can be read out without bias. However, in many situations, multiple stimuli are simultaneously present; for example, multiple motion patterns might overlap. Here we find that when multiple stimuli that overlap in their neural representation are simultaneously encoded in the population, biases in the read-out emerge. Although the bias disappears in the absence of noise, the bias is remarkably persistent at low noise levels. The bias can be reduced by competitive encoding schemes or by employing complex decoders. To study the origin of the bias, we develop a novel general framework based on gaussian processes that allows an accurate calculation of the estimate distributions of maximum likelihood decoders, and reveals that the distribution of estimates is bimodal for overlapping stimuli. The results have implications for neural coding and behavioral experiments on, for instance, overlapping motion patterns.

Distributed and Dynamic Neural Encoding of Multiple Motion Directions of Transparently Moving Stimuli in Cortical Area MT

Segmenting visual scenes into distinct objects and surfaces is a fundamental visual function. To better understand the underlying neural mechanism, we investigated how neurons in the middle temporal cortex (MT) of macaque monkeys represent overlapping random-dot stimuli moving transparently in slightly different directions. It has been shown that the neuronal response elicited by two stimuli approximately follows the average of the responses elicited by the constituent stimulus components presented alone. In this scheme of response pooling, the ability to segment two simultaneously presented motion directions is limited by the width of the tuning curve to motion in a single direction. We found that, although the population-averaged neuronal tuning showed response averaging, subgroups of neurons showed distinct patterns of response tuning and were capable of representing component directions that were separated by a small angle--less than the tuning width to unidirectional stimuli. One group of neurons preferentially represented the component direction at a specific side of the bidirectional stimuli, weighting one stimulus component more strongly than the other. Another group of neurons pooled the component responses nonlinearly and showed two separate peaks in their tuning curves even when the average of the component responses was unimodal. We also show for the first time that the direction tuning of MT neurons evolved from initially representing the vector-averaged direction of slightly different stimuli to gradually representing the component directions. Our results reveal important neural processes underlying image segmentation and suggest that information about slightly different stimulus components is computed dynamically and distributed across neurons.

SIGNIFICANCE STATEMENT

Natural scenes often contain multiple entities. The ability to segment visual scenes into distinct objects and surfaces is fundamental to sensory processing and is crucial for generating the perception of our environment. Because cortical neurons are broadly tuned to a given visual feature, segmenting two stimuli that differ only slightly is a challenge for the visual system. In this study, we discovered that many neurons in the visual cortex are capable of representing individual components of slightly different stimuli by selectively and nonlinearly pooling the responses elicited by the stimulus components. We also show for the first time that the neural representation of individual stimulus components developed over a period of ∼70-100 ms, revealing a dynamic process of image segmentation.

Neural mechanisms of speed perception: transparent motion

Visual motion on the macaque retina is processed by direction- and speed-selective neurons in extrastriate middle temporal cortex (MT). There is strong evidence for a link between the activity of these neurons and direction perception. However, there is conflicting evidence for a link between speed selectivity of MT neurons and speed perception. Here we study this relationship by using a strong perceptual illusion in speed perception: when two transparently superimposed dot patterns move in opposite directions, their apparent speed is much larger than the perceived speed of a single pattern moving at that physical speed. Moreover, the sensitivity for speed discrimination is reduced for such bidirectional patterns. We first confirmed these behavioral findings in human subjects and extended them to a monkey subject. Second, we determined speed tuning curves of MT neurons to bidirectional motion and compared these to speed tuning curves for unidirectional motion. Consistent with previous reports, the response to bidirectional motion was often reduced compared with unidirectional motion at the preferred speed. In addition, we found that tuning curves for bidirectional motion were shifted to lower preferred speeds. As a consequence, bidirectional motion of some speeds typically evoked larger responses than unidirectional motion. Third, we showed that these changes in neural responses could explain changes in speed perception with a simple labeled line decoder. These data provide new insight into the encoding of transparent motion patterns and provide support for the hypothesis that MT activity can be decoded for speed perception with a labeled line model.

Modelling adaptation to directional motion using the Adelson-Bergen energy sensor

The motion energy sensor has been shown to account for a wide range of physiological and psychophysical results in motion detection and discrimination studies. It has become established as the standard computational model for retinal movement sensing in the human visual system. Adaptation effects have been extensively studied in the psychophysical literature on motion perception, and play a crucial role in theoretical debates, but the current implementation of the energy sensor does not provide directly for modelling adaptation-induced changes in output. We describe an extension of the model to incorporate changes in output due to adaptation. The extended model first computes a space-time representation of the output to a given stimulus, and then a RC gain-control circuit ("leaky integrator") is applied to the time-dependent output. The output of the extended model shows effects which mirror those observed in psychophysical studies of motion adaptation: a decline in sensor output during stimulation, and changes in the relative of outputs of different sensors following this adaptation.

Spatial summation in macaque parietal area 7a follows a winner-take-all rule

While neurons in posterior parietal cortex have been found to signal the presence of a salient stimulus among multiple items in a display, spatial summation within their receptive field in the absence of an attentional bias has never been investigated. This information, however, is indispensable when one investigates the mechanisms of spatial attention and competition between multiple visual objects. To examine the spatial summation rule in parietal area 7a neurons, we trained rhesus monkeys to fixate on a central cross while two identical stimuli were briefly displayed in a neuron's receptive field. The response to a pair of dots was compared with the responses to the same dots when they were presented individually. The scaling and power parameters of a generalized summation algorithm varied greatly, both across neurons and across combinations of stimulus locations. However, the averaged response of the recorded population of 7a neurons was consistent with a winner-take-all rule for spatial summation. A control experiment where a monkey covertly attended to both stimuli simultaneously suggests that attention introduces additional competition by facilitating the less optimal stimulus. Thus an averaging stage is introduced between ∼ 200 and 300 ms of the response to a pair of stimuli. In short, the summation algorithm over the population of area 7a neurons carries the signature of a winner-take-all operation, with spatial attention possibly influencing the temporal dynamics of stimulus competition, that is the moment that the "winner" takes "victory" over the "loser" stimulus.

Perceptual costs for motion transparency evaluated by two performance measures

Transparency perception is recognized as one of the important phenomena to understand the computational mechanism of early visual system. Transparency perception indicates that a simple theory reconstructing a single-valued field of a visual attribute, such as an optical-flow field, cannot model the neural mechanism for the human visual system and raises a fundamental issue of how visual attributes are represented and detected in the brain. It is considered that one of the important cues to reveal the neural encoding mechanism for overlapping surfaces is the perceptual cost in transparency perception. It has been known that the perceptual performance in motion transparency is worse than that expected from single motion perception. This perceptual "cost" would reflect the encoding strategy for transparent motions. Here we present a systematic study comparing the perceptual costs in motion transparency evaluated by two performance measures. The result showed that the properties of the perceptual costs varied with the performance measures. The perceptual cost evaluated by the motion detection threshold became smaller as a directional difference between overlapping motions increased, whereas the cost examined with the precision of directional judgments became worse. A computational analysis suggests that these contradictory results cannot be explained by a simple population coding model for motion directions.

Opponent-motion mechanisms are self-normalizing

In the ultimate stage of the Adelson-Bergen motion energy model [Adelson, E. H., & Bergen, J. (1985). Spatiotemporal energy models for the perception of motion. Journal of the Optical Society of America, 2, 284-299], motion is derived from the difference between directionally opponent energies E(L) and E(R). However, Georgeson and Scott-Samuel [Georgeson, M. A., & Scott-Samuel, N. E. (1999). Motion contrast: A new metric for direction discrimination. Vision Research, 39, 4393-4402] demonstrated that motion contrast-a metric that normalizes opponent motion energy (E(L)-E(R)) by flicker energy (E(L)+E(R))-is a better descriptor of human direction discrimination. In a previous study [Rainville, S. J. M., Makous, W. L., & Scott-Samuel, N. E. (2002). The spatial properties of opponent-motion normalization. Vision Research, 42, 1727-1738], we used a lateral masking paradigm to show that opponent-motion normalization is selective for flicker position, orientation, and spatial-frequency. In the present study, we used a superposition masking paradigm and compared results to lateral masking data, as the two masking types activate local and remote normalization mechanisms differentially. Although selectivity for flicker orientation and spatial frequency varied across observers, bandwidths were similar across lateral and superimposed masking conditions. Additional experiments demonstrated that normalization signals are pooled over a spatial region whose aspect ratio and size are consistent with those of local motion detectors. Together, results show no evidence of remote normalization signals predicted by broadband inhibitory models [(e.g.) Heeger, D. J. (1992). Normalization of cell responses in cat striate cortex. Visual Neuroscience, 9, 181-197; Foley, J. M. (1994). Human luminance pattern-vision mechanisms: Masking experiments require a new model. Journal of the Optical Society of America A-Optics and Image Science, 11, 1710-1719] but support a local normalization process whose spatial properties are inherited from low-level motion detectors.

Nonlinearity of the population activity to transparent motion

How to represent transparent motion with neuronal populations is important problem for the theory of multiple motion detection. Previous models are based on the assumption that the population activity to transparent motion is proportional to a linear combination of the responses to individual motions. However, there is a possibility that the population activity becomes a nonlinear combination of each motion's component due to the interference, or cross-talk, between two moving patterns. Here we show the model analysis of how a neuronal population represents multiple motions with the spatiotemporal energy model. The model analysis indicates there is a special case that the interference leads to the nonlinearity in the population response, although the linear combination assumption is satisfied in general. This special case corresponds to locally-paired-dot (LPD) stimuli that produce no transparency. Computer simulations show that a simple model for motion detection fails to discriminate two overlapping motions in this case due to the nonlinearity in population responses, and this failure is similar to human perception in LPD stimuli. This result suggests that non-transparency perception in LPD stimuli is naturally explained by the nonlinear property of neuronal responses.

Inference and computation with population codes

In the vertebrate nervous system, sensory stimuli are typically encoded through the concerted activity of large populations of neurons. Classically, these patterns of activity have been treated as encoding the value of the stimulus (e.g., the orientation of a contour), and computation has been formalized in terms of function approximation. More recently, there have been several suggestions that neural computation is akin to a Bayesian inference process, with population activity patterns representing uncertainty about stimuli in the form of probability distributions (e.g., the probability density function over the orientation of a contour). This paper reviews both approaches, with a particular emphasis on the latter, which we see as a very promising framework for future modeling and experimental work.

Influence of viewing distance on aftereffects of moving random pixel arrays

Viewing-distance invariance of visual perception has evolutionary advantages, but it is of necessity limited by spatial and temporal resolution. Even within these resolution limits viewing-distance invariance might not be perfect or even good, but there are remarkably few studies of its precise limits. Here we ask to what extent viewing-distance invariance holds for motion aftereffects (MAEs). There are (at least) two different MAEs: one can be seen on a static test pattern (sMAE) and is tuned to low speeds, the other only becomes manifest on a dynamic noise test stimulus (dMAE) and is sensitive to higher adaptation speeds. We show that each of these MAEs has a limited viewing-distance invariance, the dMAE only for higher screen-speeds and the sMAE only for lower screen-speeds. In both cases upper angular-speed limits shift to higher values for smaller viewing-distances (lower spatial frequencies, larger fields). This upper limit is constant, independent of viewing distance, if expressed in terms of screen-speed. On the other hand the lower speed limit is fixed in angular-speed and variable in screen-speed terms. Explanations for these findings are provided. We show that there is no fixed optimum viewing-distance or optimum angular stimulus-size for either of the two MAEs.

Computational model for neural representation of multiple disparities

It has been known that the visual system can detect more than one disparity and/or motion direction at the same region in the image. These multiple (or transparent) surfaces can be perceived when, for example, we are looking scenes through a glass. However, many conventional models cannot deal with these multiple surfaces. The present paper investigates the neural encoding and decoding of multiple disparities with the binocular energy model, which is known as a biologically plausible model. Based on the analysis of the response of the energy model to multiple disparities, the present paper proposes a stereo model that can detect disparities of two overlapping surfaces.

The spatial properties of opponent-motion normalization

The final stage of the Adelson-Bergen model [J. Opt. Soc. Am. A 2 (1985) 284] computes net motion as the difference between directionally opposite energies E(L) and E(R). However, Georgeson and Scott-Samuel [Vis. Res. 39 (1999) 4393] found that human direction discrimination is better described by motion contrast (C(m))--a metric where opponent energy (E(L)-E(R)) is divided by flicker energy (E(L)+E(R)). In the present paper, we used a lateral masking paradigm to investigate the spatial properties of flicker energy involved in the normalization of opponent energy. Observers discriminated between left and right motion while viewing a checkerboard in which half of the checks contained a drifting sinusoid and the other half contained flicker (i.e. a counterphasing sinusoid). The relative luminance contrasts of flicker and motion checks determined the checkerboard's overall motion contrast C(m). We obtained selectivity functions for opponent-motion normalization by measuring C(m) thresholds whilst varying the orientation, spatial frequency, or size of flicker checks. In all conditions, performance (percent correct) decayed lawfully as we decreased motion contrast, validating the C(m) metric for our stimuli. Thresholds decreased with check size and also improved as we increased either the orientation or spatial-frequency difference between motion and flicker checks. Our data are inconsistent with Heeger-type normalization models [Vis. Neurosci. 9 (1992) 181] in which excitatory inputs are normalized by a non-selective pooling of inhibitory inputs, but data are consistent with the implicit assumption in Georgeson and Scott-Samuel's model that flicker normalization is localized in orientation, scale, and space. However, our lateral masking paradigm leaves open the possibility that the spatial properties of flicker normalization would be different if opponent and flicker energies spatially overlapped. Further characterization of motion contrast will require models of the spatial, temporal, and joint space-time properties of mechanisms mediating opponent-motion and flicker normalization.

Information processing with population codes

Information is encoded in the brain by populations or clusters of cells, rather than by single cells. This encoding strategy is known as population coding. Here we review the standard use of population codes for encoding and decoding information, and consider how population codes can be used to support neural computations such as noise removal and nonlinear mapping. More radical ideas about how population codes may directly represent information about stimulus uncertainty are also discussed.

Seeing multiple directions of motion-physiology and psychophysics

Dot patterns sliding transparently across one another are normally perceived as independently moving surfaces. Recordings from direction-selective neurons in area MT of the macaque suggested that this perceptual segregation did not depend on the presence of two peaks in the population activity. Rather, the visual system seemed to use overall shape of the population response to determine the number and directions of motion components. This approach explained a number of perceptual phenomena, including susceptibility of the motion system to direction metamers, motion patterns combining three or five directions incorrectly perceived by subjects as comprising only two directions. Our findings offer insights into the coding of multi-valued sensory signals and provide constraints for biologically based computational models.

Responses of MT and MST neurons to one and two moving objects in the receptive field

To test the effects of complex visual motion stimuli on the responses of single neurons in the middle temporal visual area (MT) and the medial superior temporal area (MST) of the macaque monkey, we compared the response elicited by one object in motion through the receptive field with the response of two simultaneously presented objects moving in different directions through the receptive field. There was an increased response to a stimulus moving in a direction other than the best direction when it was paired with a stimulus moving in the best direction. This increase was significant for all directions of motion of the non-best stimulus and the magnitude of the difference increased as the difference in the directions of the two stimuli increased. Similarly, there was a decreased response to a stimulus moving in a non-null direction when it was paired with a stimulus moving in the null direction. This decreased response in MT did not reach significance unless the second stimulus added to the null direction moved in the best direction, whereas in MST the decrease was significant when the second stimulus direction differed from the null by 90 degrees or more. Further analysis showed that the two-object responses were better predicted by taking the averaged response of the neuron to the two single-object stimuli than by summation, multiplication, or vector addition of the responses to each of the two single-object stimuli. Neurons in MST showed larger modulations than did neurons in MT with stimuli moving in both the best direction and in the null direction and the average better predicted the two-object response in area MST than in area MT. This indicates that areas MT and MST probably use a similar integrative mechanisms to create their responses to complex moving visual stimuli, but that this mechanism is further refined in MST. These experiments show that neurons in both MT and MST integrate the motion of all directions in their responses to complex moving stimuli. These results with the motion of objects were in sound agreement with those previously reported with the use of random dot patterns for the study of transparent motion in MT and suggest that these neurons use similar computational mechanisms in the processing of object and global motion stimuli.