An illusory transformation in a model of optic flow processing

Attention affects the perception of self-motion direction from optic flow

Many studies have demonstrated that attention affects the perception of many visual features. However, previous studies show conflicting results regarding the effect of attention on the perception of self-motion direction (i.e., heading) from optic flow. To address this question, we conducted three behavioral experiments and found that estimation accuracies of large headings (>14°) decreased with attention load, discrimination thresholds of these headings increased with attention load, and heading estimates were systematically compressed toward the focus of attention. Therefore, the current study demonstrated that attention affected heading perception from optic flow, showing that the perception is both information-driven and cognitive.

A model of how depth facilitates scene-relative object motion perception

Many everyday interactions with moving objects benefit from an accurate perception of their movement. Self-motion, however, complicates object motion perception because it generates a global pattern of motion on the observer’s retina and radically influences an object’s retinal motion. There is strong evidence that the brain compensates by suppressing the retinal motion due to self-motion, however, this requires estimates of depth relative to the object—otherwise the appropriate self-motion component to remove cannot be determined. The underlying neural mechanisms are unknown, but neurons in brain areas MT and MST may contribute given their sensitivity to motion parallax and depth through joint direction, speed, and disparity tuning. We developed a neural model to investigate whether cells in areas MT and MST with well-established neurophysiological properties can account for human object motion judgments during self-motion. We tested the model by comparing simulated object motion signals to human object motion judgments in environments with monocular, binocular, and ambiguous depth. Our simulations show how precise depth information, such as that from binocular disparity, may improve estimates of the retinal motion pattern due the self-motion through increased selectivity among units that respond to the global self-motion pattern. The enhanced self-motion estimates emerged from recurrent feedback connections in MST and allowed the model to better suppress the appropriate direction, speed, and disparity signals from the object’s retinal motion, improving the accuracy of the object’s movement direction represented by motion signals.

Research has shown that the accuracy with which humans perceive object motion during self-motion improves in the presence of stereo cues. Using a neural modelling approach, we explore whether this finding can be explained through improved estimation of the retinal motion induced by self-motion. Our results show that depth cues that provide information about scene structure may have a large effect on the specificity with which the neural mechanisms for motion perception represent the visual self-motion signal. This in turn enables effective removal of the retinal motion due to self-motion when the goal is to perceive object motion relative to the stationary world. These results reveal a hitherto unknown critical function of stereo tuning in the MT-MST complex, and shed important light on how the brain may recruit signals from upstream and downstream brain areas to simultaneously perceive self-motion and object motion.

Receptive field dynamics underlying MST neuronal optic flow selectivity

Optic flow informs moving observers about their heading direction. Neurons in monkey medial superior temporal (MST) cortex show heading selective responses to optic flow and planar direction selective responses to patches of local motion. We recorded MST neuronal responses to a 90 x 90 degrees optic flow display and to a 3 x 3 array of local motion patches covering the same area. Our goal was to test the hypothesis that the optic flow responses reflect the sum of the local motion responses. The local motion responses of each neuron were modeled as mixtures of Gaussians, combining the effects of two Gaussian response functions derived using a genetic algorithm, and then used to predict that neuron's optic flow responses. Some neurons showed good correspondence between local motion models and optic flow responses, others showed substantial differences. We used the genetic algorithm to modulate the relative strength of each local motion segment's responses to accommodate interactions between segments that might modulate their relative efficacy during co-activation by global patterns of optic flow. These gain modulated models showed uniformly better fits to the optic flow responses, suggesting that coactivation of receptive field segments alters neuronal response properties. We tested this hypothesis by simultaneously presenting local motion stimuli at two different sites. These two-segment stimuli revealed that interactions between response segments have direction and location specific effects that can account for aspects of optic flow selectivity. We conclude that MST's optic flow selectivity reflects dynamic interactions between spatially distributed local planar motion response mechanisms.

A neural model of how the brain computes heading from optic flow in realistic scenes

Visually-based navigation is a key competence during spatial cognition. Animals avoid obstacles and approach goals in novel cluttered environments using optic flow to compute heading with respect to the environment. Most navigation models try either explain data, or to demonstrate navigational competence in real-world environments without regard to behavioral and neural substrates. The current article develops a model that does both. The ViSTARS neural model describes interactions among neurons in the primate magnocellular pathway, including V1, MT(+), and MSTd. Model outputs are quantitatively similar to human heading data in response to complex natural scenes. The model estimates heading to within 1.5 degrees in random dot or photo-realistically rendered scenes, and within 3 degrees in video streams from driving in real-world environments. Simulated rotations of less than 1 degrees /s do not affect heading estimates, but faster simulated rotation rates do, as in humans. The model is part of a larger navigational system that identifies and tracks objects while navigating in cluttered environments.

Evidence for flow-parsing in radial flow displays

Retinal motion of objects is not in itself enough to signal whether or how objects are moving in the world; the same pattern of retinal motion can result from movement of the object, the observer or both. Estimation of scene-relative movement of an object is vital for successful completion of many simple everyday tasks. Recent research has provided evidence for a neural flow-parsing mechanism which uses the brain's sensitivity to optic flow to separate retinal motion signals into those components due to observer movement and those due to the movement of objects in the scene. In this study we provide further evidence that flow-parsing is implicated in the assessment of object trajectory during observer movement. Furthermore, it is shown that flow-parsing involves a global analysis of retinal motion, as might be expected if optic flow processing underpinned this mechanism.

Resolution of complex motion detectors in the central and peripheral visual field

We examine how local direction signals are combined to compute the focus of radial motion (FRM) in random dot patterns and examine how this process changes across the visual field. Equivalent noise analysis showed that a loss in FRM accuracy was largely attributable to an increase in local motion detector noise with little or no change in efficiency across the visual field. The minimum separation for discriminating the foci of two overlapping optic flow patterns increased in the periphery faster than predicted from the resolution for a single FRM. This behavior requires that observers average numerous local velocities to estimate the FRM, which enables resistance to internal and external noise and endows the system with the property of position invariance. However, such pooling limits the precision with which multiple looming objects can be discriminated, especially in the peripheral visual field.

An illusory transformation of optic flow fields without local motion interactions

The focus of expansion (FOE) of a radially expanding optic flow pattern that is overlapped by unidirectional laminar flow is perceptually displaced in the direction of that laminar flow. There is continuing debate on whether this effect is due to local or global motion interactions. Here, we show psychophysically that under conditions without local motion transparency the illusion becomes weaker but can still be observed. In our experiments, the radial and laminar-flow fields were not presented with overlap but separately to the left and right halves of the visual field with a blank vertical strip of 15 degrees horizontal width in between. The illusory shift observed in this condition cannot be explained by local motion interactions because (a) no transparent motion was present in the stimulus, and (b) the receptive fields of cortical cells involved in the analysis of local motion cross the vertical midline of the visual field to a limited extent. We conclude that global motion detectors that integrate motion from both halves of the visual field play a role in shifting the perceived position of the FOE and that local motion interactions may be sufficient, but are not necessary for the optic flow illusion to occur.

Spatial interference among moving targets

Peripheral vision for static form is limited both by reduced spatial acuity and by interference among adjacent features ('crowding'). However, the visibility of acuity-corrected image motion is relatively constant across the visual field. We measured whether spatial interference among nearby moving elements is similarly invariant of retinal eccentricity and assessed if motion integration could account for any observed sensitivity loss. We report that sensitivity to the direction of motion of a central target-highly visible in isolation-was strongly impaired by four drifting flanking elements. The extent of spatial interference increased with eccentricity. Random-direction flanks and flanks whose directions formed global patterns of rotation or expansion were more disruptive than flanks forming global patterns of translation, regardless of the relative direction of the target element. Spatial interference was low-pass tuned for spatial frequency and broadly tuned for temporal frequency. We show that these results challenge the generality of models of spatial interference that are based on retinal image quality, masking, confusions between target and flanks, attentional resolution limits or (simple) "averaging" of element parameters. Instead, the results suggest that spatial interference is a consequence of the integration of meaningful image structure within large receptive fields. The underlying connectivity of this integration favours low spatial frequency structure but is broadly tuned for speed.

Computational analyses for illusory transformations in the optic flow field and heading perception in the presence of moving objects

When we see a stimulus of a radial flow field (the target flow) overlapped with a lateral flow field or another radial flow field, the focus of expansion (FOE) of the target radial flow appears to be shifted in a direction. Royden and Conti [(2003). A model using MT-like motion-opponent operators explains an illusory transformation in the optic flow field. Vision Research, 43, 2811-2826] argued that local motion subtraction is crucial for explanation of this phenomenon. The flow field which causes the illusory displacement of FOE was computationally analyzed. It was shown that the flow field is approximately a rigid-motion flow; the flow can be generated by simulating a situation where an observer moves toward a stationary scene. The heading direction for the observer corresponds to the perceived position of the FOE of the radial flow pattern. It implies that any algorithms which assume rigidity of the scene and recover veridical heading explain the bias in perceived FOE. There is no need for local motion subtraction in order to explain the phenomena. Furthermore, the flow for an observer's translation in the presence of objects moving laterally or in depth was computationally analyzed. It was found that algorithms which minimizes standard error functions with less weights to the independently moving objects show similar biases in recovered heading to the bias of human observers. It implies that local motion subtraction is not necessary for explanation of the bias in perceived heading due to an object moving laterally or in depth, contrary to the argument of Royden [(2002). Computing heading in the presence of moving objects: a model that uses motion-opponent operators. Vision Research, 42, 3043-3058].

Effects of spatial arrangement of visual stimulus on inverted self-motion perception induced by the foreground motion: examination of OKN-suppression hypothesis

Our previous study revealed that a slowly moving foreground, which is presented in front of a fast-moving orthogonal background, can induce self-motion perception in the same direction as its motion (inverted vection; Vis. Res. 40 (2000) 2915). The present study shows that inverted vection becomes stronger in the conditions where the foreground stimulus is presented in the central area of observer's visual field and the observer's eyes converge on the same depth plane. These stimulus conditions are consistent with the one where the foreground can induce observer's optokinetic nystagmus more effectively, and therefore, the results of this study support our hypothesis in that mis-registered eye-movement information caused by the suppression of optokinetic nystagmus induced by the foreground motion is a critical factor in perceiving inverted vection.

A model using MT-like motion-opponent operators explains an illusory transformation in the optic flow field

Previous studies have shown that a physiologically based model using motion-opponent operators to compute heading performs accurately for simulated observer translations. Here we show how this model can explain an illusory shift in the perceived focus of expansion of a radial flow field that occurs when a field of laterally moving dots is superimposed on a field of radially moving dots. Furthermore, we can use the model to predict the perceptual shift of the focus of expansion for novel visual stimuli. These results support the hypothesis that this illusion results from motion subtraction during the processing of optic flow fields.

Compound self-motion perception induced by two kinds of optical motion

Two kinds of flow patterns consisting of random dots were presented simultaneously to subjects to investigate whether or not two kinds of vection occur simultaneously. One pattern induces vertical linear self-translation, whereas the other induces self-rotation around a vertical axis (when either pattern is presented alone). Three sets of conditions were tested. The first condition was one in which random dots moved in a summed direction of both flow vectors. In the second condition, both flow patterns were simply overlaid, whereas in the third condition, the two kinds of flow patterns were overlaid with a depth separation produced by binocular disparity. The subjects perceived both kinds of vection simultaneously in directions opposite to those of the corresponding flow components under the first condition, whereas either vection occurred mainly under the second condition. Under the third condition, both of the flows induced each kind of vection simultaneously, despite there being no physical vector summation of dot motion. The background flow induced vection in a direction opposite to the flow direction, whereas the foreground flow induced vection in the same direction as the flow direction. These results show that induced self-translation and induced self-rotation can occur simultaneously in two ways.

Computing heading in the presence of moving objects: a model that uses motion-opponent operators

Psychophysical experiments have shown that human heading judgments can be biased by the presence of moving objects. Here we present a theoretical argument that motion differences can account for the direction of bias seen in humans. We further examine the responses of a computer simulation of a model for computing heading that uses motion-opponent operators similar to cells in the primate middle temporal visual area. When moving objects are present, this model shows similar biases to those seen with humans, suggesting that such a model may underlie human heading computations.

Comparison of the spatial-frequency selectivity of local and global motion detectors

Convergent physiological and behavioral evidence indicates that the initial receptive fields responsible for motion detection are spatially localized. Consequently, the perception of global patterns of movement (such as expansion) requires that the output of these local mechanisms be integrated across visual space. We have differentiated local and global motion processes, with mixtures of coherent and incoherent moving patterns composed of bandpass filtered dots, and have measured their spatial-frequency selectivity. We report that local motion detectors show narrow-band spatial-frequency tuning (i.e., they respond only to a narrow range of spatial frequencies) but that global motion detectors show broadband spatial-frequency tuning (i.e., they integrate across a broad range of spatial frequencies), with a preference for low spatial frequencies.

A laterally interconnected neural architecture in MST accounts for psychophysical discrimination of complex motion patterns

The complex patterns of visual motion formed across the retina during self-motion, often referred to as optic flow, provide a rich source of information describing our dynamic relationship within the environment. Psychophysical studies indicate the existence of specialized detectors for component motion patterns (radial, circular, planar) that are consistent with the visual motion properties of cells in the medial superior temporal area (MST) of nonhuman primates. Here we use computational modeling and psychophysics to investigate the structural and functional role of these specialized detectors in performing a graded motion pattern (GMP) discrimination task. In the psychophysical task perceptual discrimination varied significantly with the type of motion pattern presented, suggesting perceptual correlates to the preferred motion bias reported in MST. Simulated perceptual discrimination in a population of independent MST-like neural responses showed inconsistent psychophysical performance that varied as a function of the visual motion properties within the population code. Robust psychophysical performance was achieved by fully interconnecting neural populations such that they inhibited nonpreferred units. Taken together, these results suggest that robust processing of the complex motion patterns associated with self-motion and optic flow may be mediated by an inhibitory structure of neural interactions in MST.

How stereovision interacts with optic flow perception: neural mechanisms

Optic flow, the global visual motion experienced during self-movement, supplies important navigational information. Optic flow analysis in the visual system is aided by several other visual and non-visual signals. Recent psychophysical findings demonstrate an interaction of optic flow perception and stereoscopic depth vision. Retinal disparity strongly affects an optic flow illusion, which can be related to the mechanisms of visual self-motion detection. To investigate the neuronal basis of this interaction, we tested several hypotheses by introducing different disparity contributions in a detailed neurobiological model of optic flow processing in monkey cortex. The disparity-dependent modification, which accounted best for the data suggests a specific contribution of a subset of stereoscopically modulated cortical neurons present in areas MT and MST.

Perception of self-motion from visual flow

Accurate and efficient control of self-motion is an important requirement for our daily behavior. Visual feedback about self-motion is provided by optic flow. Optic flow can be used to estimate the direction of self-motion ('heading') rapidly and efficiently. Analysis of oculomotor behavior reveals that eye movements usually accompany self-motion. Such eye movements introduce additional retinal image motion so that the flow pattern on the retina usually consists of a combination of self-movement and eye movement components. The question of whether this 'retinal flow' alone allows the brain to estimate heading, or whether an additional 'extraretinal' eye movement signal is needed, has been controversial. This article reviews recent studies that suggest that heading can be estimated visually but extraretinal signals are used to disambiguate problematic situations. The dorsal stream of primate cortex contains motion processing areas that are selective for optic flow and self-motion. Models that link the properties of neurons in these areas to the properties of heading perception suggest possible underlying mechanisms of the visual perception of self-motion.

Optic flow illusion and single neuron behaviour reconciled by a population model

Radial patterns of optic flow contain a centre of expansion that indicates the observer's direction of self-movement. When the radial pattern is viewed with transparently overlapping unidirectional motion, the centre of expansion appears to shift in the direction of the unidirectional motion [Duffy, C.J. & Wurtz, R.H. (1993) Vision Res., 33, 1481-1490]. Neurons in the medial superior temporal (MST) area of monkey cerebral cortex are thought to mediate optic flow analysis, but they do not shift their responses to parallel the illusion created by transparent overlap. The population-based model of optic flow analysis proposed by Lappe and Rauschecker replicates the illusory shift observed in perceptual studies [Lappe, M. & Rauschecker, J.P. (1995) Vision Res., 35, 1619-1631]. We analysed the behaviour of constituent neurons in the model, to gain insight into neuronal mechanisms underlying the illusion. Single model neurons did not show the illusory shift but rather graded variations of their response specificity. The shift required the aggregate response of the population. We compared the model's predictions about the behaviour of single neurons with the responses recorded from area MST. The predicted distribution of overlap effects agreed with that observed in area MST. The success of the population-based model in predicting the illusion and the neuronal behaviour suggests that area MST uses the graded responses of single neurons to create a population response that supports optic flow perception.

Emulating the visual receptive-field properties of MST neurons with a template model of heading estimation

We have proposed previously a computational neural-network model by which the complex patterns of retinal image motion generated during locomotion (optic flow) can be processed by specialized detectors acting as templates for specific instances of self-motion. The detectors in this template model respond to global optic flow by sampling image motion over a large portion of the visual field through networks of local motion sensors with properties similar to those of neurons found in the middle temporal (MT) area of primate extrastriate visual cortex. These detectors, arranged within cortical-like maps, were designed to extract self-translation (heading) and self-rotation, as well as the scene layout (relative distances) ahead of a moving observer. We then postulated that heading from optic flow is directly encoded by individual neurons acting as heading detectors within the medial superior temporal (MST) area. Others have questioned whether individual MST neurons can perform this function because some of their receptive-field properties seem inconsistent with this role. To resolve this issue, we systematically compared MST responses with those of detectors from two different configurations of the model under matched stimulus conditions. We found that the characteristic physiological properties of MST neurons can be explained by the template model. We conclude that MST neurons are well suited to support self-motion estimation via a direct encoding of heading and that the template model provides an explicit set of testable hypotheses that can guide future exploration of MST and adjacent areas within the superior temporal sulcus.

Heading detection using motion templates and eye velocity gain fields

Eye or head rotation would influence perceived heading direction if it were coded by cells tuned only to retinal flow patterns that correspond to linear self-movement. We propose a model for heading detection based on motion templates that are also Gaussian-tuned to the amount of rotational flow. Such retinal flow templates allow explicit use of extra-retinal signals to create templates tuned to head-centric flow as seen by the stationary eye. Our model predicts an intermediate layer of 'eye velocity gain fields' in which 'rate-coded' eye velocity is multiplied with responses of templates sensitive to specific retinal flow patterns. By combination of the activities of one retinal flow template and many units with an eye velocity gain field, a new type of unit appears: its preferred retinal flow changes dynamically in accordance with the eye rotation velocity. This unit's activity becomes thereby approximately invariant to the amount of eye rotation. The units with eye velocity gain fields from the motion-analogue of the units with eye position gain fields found in area 7a, which according to our general approach, are needed to transform position from retino-centric to head-centric coordinates. The rotation-tuned templates can also provide rate-coded visual estimates of eye rotation to allow a pure visual compensation for rotational flow. Our model is consistent with psychophysical data that indicate a role for extra-retinal as well as visual rotation signals in the correct perception of heading.

A model of the combination of optic flow and extraretinal eye movement signals in primate extrastriate visual cortex. Neural model of self-motion from optic flow and extraretinal cues

The determination of the direction of heading from optic flow is a complicated task. To solve it the visual system complements the optic flow by non-visual information about the occurrence of eye movements. Psychophysical studies have shown that the need for this combination depends on the structure the visual scene. In a depth-rich visual environment motion parallax can be exploited to differentiate self-translation from eye rotation. In the absence of motion parallax, i.e. in the case of movement towards a frontoparallel plane, extraretinal signals are necessary for correct heading perception ([Warren and Hannon, 1990]). [Lappe and Rauschecker (1993b)] have proposed a model of visual heading detection that reproduces many of the psychophysical findings in the absence of extraretinal input and links them to properties of single neurons in the primate visual cortex. The present work proposes a neural network model that integrates extraretinal signals into this network. The model is compared with psychophysical and neurophysiological data from experiments in human and non-human primates. The combined visual/extraretinal model reproduces human behavior in the case of movement towards a frontoparallel plane. Single model neurons exhibit several similarities to neurons from the medial superior temporal (MST) area of the macaque monkey. Similar to MST cells ([Erickson and Thier, 1991]) they differentiate between self-induced visual motion that results from eye movements in a stationary environment, and real motion in the environment. The model predicts that this differentiation can also be achieved visually, i.e. without extraretinal input. Other simulations followed experiments by [Bradley et al. (1996)], in which flow fields were presented that simulated observer translation towards a frontoparallel plane plus an eye rotation. Similar to MST cells, model neurons shift their preference for the focus of expansion along the direction of the eye movement when extraretinal input is not available. They respond to the retinal location of the focus of expansion which is shifted by the eye movement. In the presence of extraretinal input the preference for the focus of expansion is largely invariant to eye movements and tied to the location of the focus of expansion with regard to the visual scene. The model proposes that extraretinal compensation for eye movements need not be perfect in single neurons to achieve accurate heading detection. It thereby shows that the incomplete compensation found in most MST neurons is sufficient to explain the psychophysical data.

Interaction of stereo vision and optic flow processing revealed by an illusory stimulus

The influence of stereoscopic vision on the perception of optic flow fields was investigated in experiments based on a recently described illusion. In this illusion, subjects perceive a shift of the center of an expanding optic flow field when it is transparently superimposed by a unidirectional motion pattern. This illusory shift can be explained by the visual system taking the presented flow pattern as a certain self-motion flow field. Here we examined the dependence of the illusory transformation on differences in depth between the two superimposed motion patterns. Presenting them with different relative binocular disparities, we found a strong variation in the magnitude of the illusory shift. Especially when translation was in front of expansion, a highly significant decrease of the illusory shift occurred, down to 25% of its magnitude at zero disparity. These findings confirm the assumption that the motion pattern is interpreted as a self-motion flow field. In a further experiment we presented monocular depth cues by changing dot size and dot density. This caused a reduction of the illusory shift which is distinctly smaller than under stereoscopic presentation. We conclude that the illusory optic flow transformation is modified by depth information, especially by binocular disparity. The findings are linked to the phenomenon of induced motion and are related to neurophysiology.

Radial motion looks faster

Current models of motion perception depend on unidirectional motion-sensitive mechanisms that provide local inputs for complex pattern motion, such as optic flow. To test the generality of such models, we asked observers to compare the speed of radial gratings with the translational speed of vertical gratings. The speed of the radial gratings was consistently overestimated by 20-60% relative to that of translating gratings that were identical in all other respects. The speed bias was not associated with a general spatial or temporal processing bias, nor with the high relative speed of points about the center of expansion/contraction. The bias increased non-linearly with the size of sectors of the radiating pattern exposed. As the motion of the two patterns was locally identical but judged differently, the apparent speed of both kinds of motion cannot be served by any mechanism, nor described by any model, that is based entirely on local motion signals. We speculate that the greater apparent speed of the radial motion has to do with apparent motion in depth.

Human heading judgments in the presence of moving objects

When moving toward a stationary scene, people judge their heading quite well from visual information alone. Much experimental and modeling work has been presented to analyze how people judge their heading for stationary scenes. However, in everyday life, we often move through scenes that contain moving objects. Most models have difficulty computing heading when moving objects are in the scene, and few studies have examined how well humans perform in the presence of moving objects. In this study, we tested how well people judge their heading in the presence of moving objects. We found that people perform remarkably well under a variety of conditions. The only condition that affects an observer's ability to judge heading accurately consists of a large moving object crossing the observer's path. In this case, the presence of the object causes a small bias in the heading judgments. For objects moving horizontally with respect to the observer, this bias is in the object's direction of motion. These results present a challenge for computational models.