Interaction of stereo vision and optic flow processing revealed by an illusory stimulus

Heading perception depends on time-varying evolution of optic flow

There is considerable support for the hypothesis that perception of heading in the presence of rotation is mediated by instantaneous optic flow. This hypothesis, however, has never been tested. We introduce a method, termed "nonvarying phase motion," for generating a stimulus that conveys a single instantaneous optic flow field, even though the stimulus is presented for an extended period of time. In this experiment, observers viewed stimulus videos and performed a forced-choice heading discrimination task. For nonvarying phase motion, observers made large errors in heading judgments. This suggests that instantaneous optic flow is insufficient for heading perception in the presence of rotation. These errors were mostly eliminated when the velocity of phase motion was varied over time to convey the evolving sequence of optic flow fields corresponding to a particular heading. This demonstrates that heading perception in the presence of rotation relies on the time-varying evolution of optic flow. We hypothesize that the visual system accurately computes heading, despite rotation, based on optic acceleration, the temporal derivative of optic flow.

Retinal Stabilization Reveals Limited Influence of Extraretinal Signals on Heading Tuning in the Medial Superior Temporal Area

Heading perception in primates depends heavily on visual optic-flow cues. Yet during self-motion, heading percepts remain stable, even though smooth-pursuit eye movements often distort optic flow. According to theoretical work, self-motion can be represented accurately by compensating for these distortions in two ways: via retinal mechanisms or via extraretinal efference-copy signals, which predict the sensory consequences of movement. Psychophysical evidence strongly supports the efference-copy hypothesis, but physiological evidence remains inconclusive. Neurons that signal the true heading direction during pursuit are found in visual areas of monkey cortex, including the dorsal medial superior temporal area (MSTd). Here we measured heading tuning in MSTd using a novel stimulus paradigm, in which we stabilize the optic-flow stimulus on the retina during pursuit. This approach isolates the effects on neuronal heading preferences of extraretinal signals, which remain active while the retinal stimulus is prevented from changing. Our results from 3 female monkeys demonstrate a significant but small influence of extraretinal signals on the preferred heading directions of MSTd neurons. Under our stimulus conditions, which are rich in retinal cues, we find that retinal mechanisms dominate physiological corrections for pursuit eye movements, suggesting that extraretinal cues, such as predictive efference-copy mechanisms, have a limited role under naturalistic conditions.SIGNIFICANCE STATEMENT Sensory systems discount stimulation caused by an animal's own behavior. For example, eye movements cause irrelevant retinal signals that could interfere with motion perception. The visual system compensates for such self-generated motion, but how this happens is unclear. Two theoretical possibilities are a purely visual calculation or one using an internal signal of eye movements to compensate for their effects. The latter can be isolated by experimentally stabilizing the image on a moving retina, but this approach has never been adopted to study motion physiology. Using this method, we find that extraretinal signals have little influence on activity in visual cortex, whereas visually based corrections for ongoing eye movements have stronger effects and are likely most important under real-world conditions.

Where are you heading? Flexible integration of retinal and extra-retinal cues during self-motion perception

As we move forward in the environment, we experience a radial expansion of the retinal image, wherein the center corresponds to the instantaneous direction of self-motion. Humans can precisely perceive their heading direction even when the retinal motion is distorted by gaze shifts due to eye/body rotations. Previous studies have suggested that both retinal and extra-retinal strategies can compensate for the retinal image distortion. However, the relative contributions of each strategy remain unclear. To address this issue, we devised a two-alternative-headings discrimination task, in which participants had either real or simulated pursuit eye movements. The two conditions had the same retinal input but either with or without extra-retinal eye movement signals. Thus, the behavioral difference between conditions served as a metric of extra-retinal contribution. We systematically and independently manipulated pursuit speed, heading speed, and the reliability of retinal signals. We found that the levels of extra-retinal contributions increased with increasing pursuit speed (stronger extra-retinal signal), and with decreasing heading speed (weaker retinal signal). In addition, extra-retinal contributions also increased as we corrupted retinal signals with noise. Our results revealed that the relative magnitude of retinal and extra-retinal contributions was not fixed but rather flexibly adjusted to each specific task condition. This task-dependent, flexible integration appears to take the form of a reliability-based weighting scheme that maximizes heading performance.

Dissociation of retinal and headcentric disparity signals in dorsal human cortex

Recent fMRI studies have shown fusion of visual motion and disparity signals for shape perception (Ban et al., 2012), and unmasking camouflaged surfaces (Rokers et al., 2009), but no such interaction is known for typical dorsal motion pathway tasks, like grasping and navigation. Here, we investigate human speed perception of forward motion and its representation in the human motion network. We observe strong interaction in medial (V3ab, V6) and lateral motion areas (MT⁺), which differ significantly. Whereas the retinal disparity dominates the binocular contribution to the BOLD activity in the anterior part of area MT⁺, headcentric disparity modulation of the BOLD response dominates in area V3ab and V6. This suggests that medial motion areas not only represent rotational speed of the head (Arnoldussen et al., 2011), but also translational speed of the head relative to the scene. Interestingly, a strong response to vergence eye movements was found in area V1, which showed a dependency on visual direction, just like vertical-size disparity. This is the first report of a vertical-size disparity correlate in human striate cortex.

Self-motion perception in the elderly

Self-motion through space generates a visual pattern called optic flow. It can be used to determine one's direction of self-motion (heading). Previous studies have already shown that this perceptual ability, which is of critical importance during everyday life, changes with age. In most of these studies subjects were asked to judge whether they appeared to be heading to the left or right of a target. Thresholds were found to increase continuously with age. In our current study, we were interested in absolute rather than relative heading judgments and in the question about a potential neural correlate of an age-related deterioration of heading perception. Two groups, older test subjects and younger controls, were shown optic flow stimuli in a virtual-reality setup. Visual stimuli simulated self-motion through a 3-D cloud of dots and subjects had to indicate their perceived heading direction after each trial. In different subsets of experiments we varied individually relevant stimulus parameters: presentation time, number of dots in the display, stereoscopic vs. non-stereoscopic stimulation, and motion coherence. We found decrements in heading performance with age for each stimulus parameter. In a final step we aimed to determine a putative neural basis of this behavioral decline. To this end we modified a neural network model which previously has proven to be capable of reproduce and predict certain aspects of heading perception. We show that the observed data can be modeled by implementing an age related neuronal cell loss in this neural network. We conclude that a continuous decline of certain aspects of motion perception, among them heading, might be based on an age-related progressive loss of groups of neurons being activated by visual motion.

Population anisotropy in area MT explains a perceptual difference between near and far disparity motion segmentation

Segmentation of the visual scene into relevant object components is a fundamental process for successfully interacting with our surroundings. Many visual cues, including motion and binocular disparity, support segmentation, yet the mechanisms using these cues are unclear. We used a psychophysical motion discrimination task in which noise dots were displaced in depth to investigate the role of segmentation through disparity cues in visual motion stimuli (experiment 1). We found a subtle, but significant, bias indicating that near disparity noise disrupted the segmentation of motion more than equidistant far disparity noise. A control experiment showed that the near-far difference could not be attributed to attention (experiment 2). To account for the near-far bias, we constructed a biologically constrained model using recordings from neurons in the middle temporal area (MT) to simulate human observers' performance on experiment 1. Performance of the model of MT neurons showed a near-disparity skew similar to that shown by human observers. To isolate the cause of the skew, we simulated performance of a model containing units derived from properties of MT neurons, using phase-modulated Gabor disparity tuning. Using a skewed-normal population distribution of preferred disparities, the model reproduced the elevated motion discrimination thresholds for near-disparity noise, whereas a skewed-normal population of phases (creating individually asymmetric units) did not lead to any performance skew. Results from the model suggest that the properties of neurons in area MT are computationally sufficient to perform disparity segmentation during motion processing and produce similar disparity biases as those produced by human observers.

An illusory transformation of optic flow fields without local motion interactions

The focus of expansion (FOE) of a radially expanding optic flow pattern that is overlapped by unidirectional laminar flow is perceptually displaced in the direction of that laminar flow. There is continuing debate on whether this effect is due to local or global motion interactions. Here, we show psychophysically that under conditions without local motion transparency the illusion becomes weaker but can still be observed. In our experiments, the radial and laminar-flow fields were not presented with overlap but separately to the left and right halves of the visual field with a blank vertical strip of 15 degrees horizontal width in between. The illusory shift observed in this condition cannot be explained by local motion interactions because (a) no transparent motion was present in the stimulus, and (b) the receptive fields of cortical cells involved in the analysis of local motion cross the vertical midline of the visual field to a limited extent. We conclude that global motion detectors that integrate motion from both halves of the visual field play a role in shifting the perceived position of the FOE and that local motion interactions may be sufficient, but are not necessary for the optic flow illusion to occur.

Global speed averaging is tuned for binocular disparity

The extent to which local speeds at different depths are averaged to determine global speed was determined using a version of the Global Dot Motion (GDM) stimulus. Judgments of the apparent speed of fast moving dots (4.05-10.53 deg/s) in the presence of slow moving dots (4.05 deg/s) were measured using GDM stimuli which simulated radially expanding motion, and which confined slow and fast speed dots exclusively to alternating wedge shaped sectors. The presence of slow moving dots in the stimulus reduced the apparent speed of high-speed dots in a manner consistent with speed averaging. However with increases in depth, produced by a difference in binocular disparity between dots in alternating sectors, speed averaging became less effective, and the relationship between speed and disparity resembled a tuning function. We discuss our results in light of research that clarifies the functional properties of global motion mechanisms in the primate cortex.

Computational analyses for illusory transformations in the optic flow field and heading perception in the presence of moving objects

When we see a stimulus of a radial flow field (the target flow) overlapped with a lateral flow field or another radial flow field, the focus of expansion (FOE) of the target radial flow appears to be shifted in a direction. Royden and Conti [(2003). A model using MT-like motion-opponent operators explains an illusory transformation in the optic flow field. Vision Research, 43, 2811-2826] argued that local motion subtraction is crucial for explanation of this phenomenon. The flow field which causes the illusory displacement of FOE was computationally analyzed. It was shown that the flow field is approximately a rigid-motion flow; the flow can be generated by simulating a situation where an observer moves toward a stationary scene. The heading direction for the observer corresponds to the perceived position of the FOE of the radial flow pattern. It implies that any algorithms which assume rigidity of the scene and recover veridical heading explain the bias in perceived FOE. There is no need for local motion subtraction in order to explain the phenomena. Furthermore, the flow for an observer's translation in the presence of objects moving laterally or in depth was computationally analyzed. It was found that algorithms which minimizes standard error functions with less weights to the independently moving objects show similar biases in recovered heading to the bias of human observers. It implies that local motion subtraction is not necessary for explanation of the bias in perceived heading due to an object moving laterally or in depth, contrary to the argument of Royden [(2002). Computing heading in the presence of moving objects: a model that uses motion-opponent operators. Vision Research, 42, 3043-3058].

Effects of spatial arrangement of visual stimulus on inverted self-motion perception induced by the foreground motion: examination of OKN-suppression hypothesis

Our previous study revealed that a slowly moving foreground, which is presented in front of a fast-moving orthogonal background, can induce self-motion perception in the same direction as its motion (inverted vection; Vis. Res. 40 (2000) 2915). The present study shows that inverted vection becomes stronger in the conditions where the foreground stimulus is presented in the central area of observer's visual field and the observer's eyes converge on the same depth plane. These stimulus conditions are consistent with the one where the foreground can induce observer's optokinetic nystagmus more effectively, and therefore, the results of this study support our hypothesis in that mis-registered eye-movement information caused by the suppression of optokinetic nystagmus induced by the foreground motion is a critical factor in perceiving inverted vection.

A model using MT-like motion-opponent operators explains an illusory transformation in the optic flow field

Previous studies have shown that a physiologically based model using motion-opponent operators to compute heading performs accurately for simulated observer translations. Here we show how this model can explain an illusory shift in the perceived focus of expansion of a radial flow field that occurs when a field of laterally moving dots is superimposed on a field of radially moving dots. Furthermore, we can use the model to predict the perceptual shift of the focus of expansion for novel visual stimuli. These results support the hypothesis that this illusion results from motion subtraction during the processing of optic flow fields.

Compound self-motion perception induced by two kinds of optical motion

Two kinds of flow patterns consisting of random dots were presented simultaneously to subjects to investigate whether or not two kinds of vection occur simultaneously. One pattern induces vertical linear self-translation, whereas the other induces self-rotation around a vertical axis (when either pattern is presented alone). Three sets of conditions were tested. The first condition was one in which random dots moved in a summed direction of both flow vectors. In the second condition, both flow patterns were simply overlaid, whereas in the third condition, the two kinds of flow patterns were overlaid with a depth separation produced by binocular disparity. The subjects perceived both kinds of vection simultaneously in directions opposite to those of the corresponding flow components under the first condition, whereas either vection occurred mainly under the second condition. Under the third condition, both of the flows induced each kind of vection simultaneously, despite there being no physical vector summation of dot motion. The background flow induced vection in a direction opposite to the flow direction, whereas the foreground flow induced vection in the same direction as the flow direction. These results show that induced self-translation and induced self-rotation can occur simultaneously in two ways.

Consistent stereoscopic information increases the perceived speed of vection in depth

Previous research found that adding stereoscopic information to radially expanding optic flow decreased vection onsets and increased vection durations (Palmisano, 1996 Perception & Psychophysics 58 1168-1176). In the current experiments, stereoscopic cues were also found to increase perceptions of vection speed and self-displacement during vection in depth--but only when these cues were consistent with monocularly available information about self-motion. Stereoscopic information did not appear to be improving vection by increasing the perceived maximum extent of displays or by making displays appear more three-dimensional. Rather, it appeared that consistent patterns of stereoscopic optic flow provided extra, purely binocular information about vection speed, which resulted in faster/more compelling illusions of self-motion in depth.

Asymmetrical masking between radial and parallel motion flow in transparent displays

Human spatial behavior remains surprisingly accurate and stable, despite the complexities of the concurrent retinal image flow. How does visual motion analysis contribute to behavioral stability? To approach this question, we rely on current knowledge about heading perception and coherent motion extraction from animated random dot displays. This chapter presents psychophysical evidence for an asymmetry in the global visual decomposition of the retinal motion flow. We examined the ability of human observers to identify the radial and the parallel flow component in displays where the two motions were transparently superimposed. Whereas sensitivity to the radial motion remained unaffected by the presence of parallel flow, sensitivity to the latter was greatly degraded in the presence of radial flow. This imbalance was observed within the first 200 ms of motion exposure, prior to the perception of heading (430 ms; Hooge et al. (1999) Exp. Brain Res., 129: 615-628). Although such an asymmetrical masking is well suited to minimize heading errors, whether it really contributes to the extraction of behavioral direction remains an open question.

How stereovision interacts with optic flow perception: neural mechanisms

Optic flow, the global visual motion experienced during self-movement, supplies important navigational information. Optic flow analysis in the visual system is aided by several other visual and non-visual signals. Recent psychophysical findings demonstrate an interaction of optic flow perception and stereoscopic depth vision. Retinal disparity strongly affects an optic flow illusion, which can be related to the mechanisms of visual self-motion detection. To investigate the neuronal basis of this interaction, we tested several hypotheses by introducing different disparity contributions in a detailed neurobiological model of optic flow processing in monkey cortex. The disparity-dependent modification, which accounted best for the data suggests a specific contribution of a subset of stereoscopically modulated cortical neurons present in areas MT and MST.

Perception of self-motion from visual flow

Accurate and efficient control of self-motion is an important requirement for our daily behavior. Visual feedback about self-motion is provided by optic flow. Optic flow can be used to estimate the direction of self-motion ('heading') rapidly and efficiently. Analysis of oculomotor behavior reveals that eye movements usually accompany self-motion. Such eye movements introduce additional retinal image motion so that the flow pattern on the retina usually consists of a combination of self-movement and eye movement components. The question of whether this 'retinal flow' alone allows the brain to estimate heading, or whether an additional 'extraretinal' eye movement signal is needed, has been controversial. This article reviews recent studies that suggest that heading can be estimated visually but extraretinal signals are used to disambiguate problematic situations. The dorsal stream of primate cortex contains motion processing areas that are selective for optic flow and self-motion. Models that link the properties of neurons in these areas to the properties of heading perception suggest possible underlying mechanisms of the visual perception of self-motion.

Dynamical use of different sources of information in heading judgments from retinal flow

The optic flow arising in the eyes of an observer during self-motion is influenced by the occurrence of eye movements. The determination of heading during eye movements may be based on the pattern of retinal image motion (the retinal flow) or on an additional use of an extraretinal eye-movement signal. Previous research has presented support for either of these hypotheses, depending on the movement geometry and the layout of the visual scene. A special situation in which all previous studies unequivocally have agreed that an extra-retinal signal is required occurs when the visual scene consists of a single frontoparallel plane. In this situation eye movements shift the center of expansion on the retina to a location that does not correspond to the direction of self-movement. Without extraretinal input, human observers confuse the center of expansion with their heading and show a systematical heading estimation error. We reexamined and further investigated this situation. We presented retinal flow stimuli on a large projection screen in the absence of extra-retinal input and varied stimulus size, presentation duration, and orientation of the plane. In contrast to previous studies we found that in the case of a perpendicular approach toward the plane, heading judgments can be accurate. Accurate judgments were observed when the field of view was large (90 degrees x 90 degrees) and the stimulus duration was short (< or = 0.5 s). For a small field of view or a prolonged stimulus presentation, a systematic and previously described error appeared that is related to the radial structure of the flow field and the location of the center of expansion. An oblique approach toward the plane results in an ambiguous flow field with two mathematically possible solutions for heading. In this situation, when the stimulus duration was short, subjects reported a perceived heading midway between these two solutions. For longer flow sequences, subjects again chose the center of expansion. Our results suggest a dynamical change in the analysis or interpretation of retinal flow during heading perception.

Optic flow illusion and single neuron behaviour reconciled by a population model

Radial patterns of optic flow contain a centre of expansion that indicates the observer's direction of self-movement. When the radial pattern is viewed with transparently overlapping unidirectional motion, the centre of expansion appears to shift in the direction of the unidirectional motion [Duffy, C.J. & Wurtz, R.H. (1993) Vision Res., 33, 1481-1490]. Neurons in the medial superior temporal (MST) area of monkey cerebral cortex are thought to mediate optic flow analysis, but they do not shift their responses to parallel the illusion created by transparent overlap. The population-based model of optic flow analysis proposed by Lappe and Rauschecker replicates the illusory shift observed in perceptual studies [Lappe, M. & Rauschecker, J.P. (1995) Vision Res., 35, 1619-1631]. We analysed the behaviour of constituent neurons in the model, to gain insight into neuronal mechanisms underlying the illusion. Single model neurons did not show the illusory shift but rather graded variations of their response specificity. The shift required the aggregate response of the population. We compared the model's predictions about the behaviour of single neurons with the responses recorded from area MST. The predicted distribution of overlap effects agreed with that observed in area MST. The success of the population-based model in predicting the illusion and the neuronal behaviour suggests that area MST uses the graded responses of single neurons to create a population response that supports optic flow perception.

Global induced motion and visual stability in an optic flow illusion

When an expansion flow field of moving dots is overlapped by planar motion, observers perceive an illusory displacement of the focus of expansion (FOE) in the direction of the planar motion (Duffy and Wurtz, Vision Research, 1993;33:1481-1490). The illusion may be a consequence of induced motion, wherein an induced component of motion relative to planar dots is added to the motions of expansion dots to produce the FOE shift. While such a process could be mediated by local 'center-surround' receptive fields, the effect could also be due to a higher level process which detects and subtracts large-field planar motion from the flow field. We probed the mechanisms underlying this illusion by adding varying amounts of rotation to the expansion stimulus, and by varying the speed and size of the planar motion field. The introduction of rotation into the stimulus produces an illusory shift in a direction perpendicular to the planar motion. Larger FOE shifts were perceived for greater speeds and sizes of planar motion fields, although the speed effect saturated at high speeds. While the illusion appears to share a common mechanism with center-surround induced motion, our results also point to involvement of a more global mechanism that subtracts coherent planar motion from the flow field. Such a process might help to maintain visual stability during eye movements.