Does the middle temporal area carry vestibular signals related to self-motion?

F = ma. Is the macaque brain Newtonian?

Intuitive Physics, the ability to anticipate how the physical events involving mass objects unfold in time and space, is a central component of intelligent systems. Intuitive physics is a promising tool for gaining insight into mechanisms that generalize across species because both humans and non-human primates are subject to the same physical constraints when engaging with the environment. Physical reasoning abilities are widely present within the animal kingdom, but monkeys, with acute 3D vision and a high level of dexterity, appreciate and manipulate the physical world in much the same way humans do.

Neurons in Primate Area MSTd Signal Eye Movement Direction Inferred from Dynamic Perspective Cues in Optic Flow

Smooth eye movements are common during natural viewing; we frequently rotate our eyes to track moving objects or to maintain fixation on an object during self-movement. Reliable information about smooth eye movements is crucial to various neural computations, such as estimating heading from optic flow or judging depth from motion parallax. While it is well established that extraretinal signals (e.g., efference copies of motor commands) carry critical information about eye velocity, the rotational optic flow field produced by eye rotations also carries valuable information. Although previous work has shown that dynamic perspective cues in optic flow can be used in computations that require estimates of eye velocity, it has remained unclear where and how the brain processes these visual cues and how they are integrated with extraretinal signals regarding eye rotation. We examined how neurons in the dorsal region of the medial superior temporal area (MSTd) of two male rhesus monkeys represent the direction of smooth pursuit eye movements based on both visual cues (dynamic perspective) and extraretinal signals. We find that most MSTd neurons have matched preferences for the direction of eye rotation based on visual and extraretinal signals. Moreover, neural responses to combinations of these signals are well predicted by a weighted linear summation model. These findings demonstrate a neural substrate for representing the velocity of smooth eye movements based on rotational optic flow and establish area MSTd as a key node for integrating visual and extraretinal signals into a more generalized representation of smooth eye movements.SIGNIFICANCE STATEMENT We frequently rotate our eyes to smoothly track objects of interest during self-motion. Information about eye velocity is crucial for a variety of computations performed by the brain, including depth perception and heading perception. Traditionally, information about eye rotation has been thought to arise mainly from extraretinal signals, such as efference copies of motor commands. Previous work shows that eye velocity can also be inferred from rotational optic flow that accompanies smooth eye movements, but the neural origins of these visual signals about eye rotation have remained unknown. We demonstrate that macaque neurons signal the direction of smooth eye rotation based on visual signals, and that they integrate both visual and extraretinal signals regarding eye rotation in a congruent fashion.

Visual motion perception as online hierarchical inference

Identifying the structure of motion relations in the environment is critical for navigation, tracking, prediction, and pursuit. Yet, little is known about the mental and neural computations that allow the visual system to infer this structure online from a volatile stream of visual information. We propose online hierarchical Bayesian inference as a principled solution for how the brain might solve this complex perceptual task. We derive an online Expectation-Maximization algorithm that explains human percepts qualitatively and quantitatively for a diverse set of stimuli, covering classical psychophysics experiments, ambiguous motion scenes, and illusory motion displays. We thereby identify normative explanations for the origin of human motion structure perception and make testable predictions for future psychophysics experiments. The proposed online hierarchical inference model furthermore affords a neural network implementation which shares properties with motion-sensitive cortical areas and motivates targeted experiments to reveal the neural representations of latent structure.

Causal contribution of optic flow signal in Macaque extrastriate visual cortex for roll perception

Optic flow is a powerful cue for inferring self-motion status which is critical for postural control, spatial orientation, locomotion and navigation. In primates, neurons in extrastriate visual cortex (MSTd) are predominantly modulated by high-order optic flow patterns (e.g., spiral), yet a functional link to direct perception is lacking. Here, we applied electrical microstimulation to selectively manipulate population of MSTd neurons while macaques discriminated direction of rotation around line-of-sight (roll) or direction of linear-translation (heading), two tasks which were orthogonal in 3D spiral coordinate using a four-alternative-forced-choice paradigm. Microstimulation frequently biased animal's roll perception towards coded labeled-lines of the artificial-stimulated neurons in either context with spiral or pure-rotation stimuli. Choice frequency was also altered between roll and translation flow-pattern. Our results provide direct causal-link evidence supporting that roll signals in MSTd, despite often mixed with translation signals, can be extracted by downstream areas for perception of rotation relative to gravity-vertical.

Unveiling Dynamic System Strategies for Multisensory Processing: From Neuronal Fixed-Criterion Integration to Population Bayesian Inference

Multisensory processing is of vital importance for survival in the external world. Brain circuits can both integrate and separate visual and vestibular senses to infer self-motion and the motion of other objects. However, it is largely debated how multisensory brain regions process such multisensory information and whether they follow the Bayesian strategy in this process. Here, we combined macaque physiological recordings in the dorsal medial superior temporal area (MST-d) with modeling of synaptically coupled multilayer continuous attractor neural networks (CANNs) to study the underlying neuronal circuit mechanisms. In contrast to previous theoretical studies that focused on unisensory direction preference, our analysis showed that synaptic coupling induced cooperation and competition in the multisensory circuit and caused single MST-d neurons to switch between sensory integration or separation modes based on the fixed-criterion causal strategy, which is determined by the synaptic coupling strength. Furthermore, the prior of sensory reliability was represented by pooling diversified criteria at the MST-d population level, and the Bayesian strategy was achieved in downstream neurons whose causal inference flexibly changed with the prior. The CANN model also showed that synaptic input balance is the dynamic origin of neuronal direction preference formation and further explained the misalignment between direction preference and inference observed in previous studies. This work provides a computational framework for a new brain-inspired algorithm underlying multisensory computation.

Cortical Mechanisms of Multisensory Linear Self-motion Perception

Accurate self-motion perception, which is critical for organisms to survive, is a process involving multiple sensory cues. The two most powerful cues are visual (optic flow) and vestibular (inertial motion). Psychophysical studies have indicated that humans and nonhuman primates integrate the two cues to improve the estimation of self-motion direction, often in a statistically Bayesian-optimal way. In the last decade, single-unit recordings in awake, behaving animals have provided valuable neurophysiological data with a high spatial and temporal resolution, giving insight into possible neural mechanisms underlying multisensory self-motion perception. Here, we review these findings, along with new evidence from the most recent studies focusing on the temporal dynamics of signals in different modalities. We show that, in light of new data, conventional thoughts about the cortical mechanisms underlying visuo-vestibular integration for linear self-motion are challenged. We propose that different temporal component signals may mediate different functions, a possibility that requires future studies.

Postural sway in the moving room scenario: New evidence for functional dissociation between self-motion perception and postural control

Postural control in quiet standing is often explained by a reflexive response to optical flow, the apparent motion of environmental objects in a visual scene. However, moving room experiments show that even small-amplitude body sway can evoke odd sensations or motion sickness, indicating that a consciousness factor may also be involved. Studies targeting perception of self-motion, vection, typically use rapid visual stimuli moving in a single direction to maintain a constant feeling of vection, and there are few studies of vection using low-speed sinusoidal visual stimuli similar to human pendular movement. In the present study we searched for changes in postural control during periods of vection during quiet standing. Participants (N = 19, age = 20.4 ±1.1 years) were shown dynamic visual stimuli in the form of sinusoidally expanding and contracting random dots, and the stimuli speed and visual field were manipulated. Posture was continually evaluated using Center of Pressure (CoP) measurements. Participants were also asked to report feelings of vection, both by pressing a button during the trial and through an overall rating at the end of each trial. Using repeated-measures ANOVA, we assessed changes in the CoP and vection variables between experimental conditions, as well as possible interactions between the variables. The results show that postural reaction and vection were both affected by the visual stimuli and varied with speed. The peripheral visual field was found to couple to stronger feeling of vection and better quality of postural control. However, no significant relationship between postural control and vection, nor evidence of vection interaction to the relationship between optical flow and postural control, was found. Based on our results we conclude that for postural stability during quiet standing, visual cues dominate over any potential consciousness factor arising due to vection.

Primate extrastriate cortical area MST: a gateway between sensation and cognition

Primate visual cortex consists of dozens of distinct brain areas, each providing a highly specialized component to the sophisticated task of encoding the incoming sensory information and creating a representation of our visual environment that underlies our perception and action. One such area is the medial superior temporal cortex (MST), a motion-sensitive, direction-selective part of the primate visual cortex. It receives most of its input from the middle temporal (MT) area, but MST cells have larger receptive fields and respond to more complex motion patterns. The finding that MST cells are tuned for optic flow patterns has led to the suggestion that the area plays an important role in the perception of self-motion. This hypothesis has received further support from studies showing that some MST cells also respond selectively to vestibular cues. Furthermore, the area is part of a network that controls the planning and execution of smooth pursuit eye movements and its activity is modulated by cognitive factors, such as attention and working memory. This review of more than 90 studies focuses on providing clarity of the heterogeneous findings on MST in the macaque cortex and its putative homolog in the human cortex. From this analysis of the unique anatomical and functional position in the hierarchy of areas and processing steps in primate visual cortex, MST emerges as a gateway between perception, cognition, and action planning. Given this pivotal role, this area represents an ideal model system for the transition from sensation to cognition.

Visual Perception of Heading in the Syndrome of Oculopalatal Tremor

Perception of our linear motion, heading, relies on convergence from multiple sensory systems utilizing visual and vestibular signals. Multisensory convergence takes place in the visuo-vestibular areas of the cerebral cortex and posterior cerebellar vermis. Latter closely connected with the inferior olive may malfunction in disorders of olivo-cerebellar hypersynchrony, such as the syndrome of oculopalatal tremor (OPT). We had recently shown an impairment in vestibular heading perception in the subjects with OPT. Here we asked whether the hypersynchrony in the inferior-olive cerebellar circuit also affects the visual perception of heading, and the impairment is coupled with the deficits in vestibular heading perception. Three subjects with OPT and 11 healthy controls performed a two-alternative forced-choice task in two separate experiments; one when they were moved en bloc in a straight-ahead forward direction or at multiple heading angles to the right or the left; and second when under virtual reality goggle they experienced the movement of star cloud leading to the percept of heading straight, left or to the right at the heading angles similar to those utilized in the vestibular task. The resultant psychometric function curves, derived from the two-alternative-forced-choice task, revealed abnormal threshold to perceive heading direction, abnormal sensitivity to the change in heading direction compared to straight ahead, and a bias towards one side. Although the impairment was present in both visual and vestibular heading perception, the deficits were not coupled.

Processing of object motion and self-motion in the lateral subdivision of the medial superior temporal area in macaques

Multiple areas of macaque cortex are involved in visual motion processing, but their relative functional roles remain unclear. The medial superior temporal (MST) area is typically divided into lateral (MSTl) and dorsal (MSTd) subdivisions that are thought to be involved in processing object motion and self-motion, respectively. Whereas MSTd has been studied extensively with regard to processing visual and nonvisual self-motion cues, little is known about self-motion signals in MSTl, especially nonvisual signals. Moreover, little is known about how self-motion and object motion signals interact in MSTl and how this differs from interactions in MSTd. We compared the visual and vestibular heading tuning of neurons in MSTl and MSTd using identical stimuli. Our findings reveal that both visual and vestibular heading signals are weaker in MSTl than in MSTd, suggesting that MSTl is less well suited to participate in self-motion perception than MSTd. We also tested neurons in both areas with a variety of combinations of object motion and self-motion. Our findings reveal that vestibular signals improve the separability of coding of heading and object direction in both areas, albeit more strongly in MSTd due to the greater strength of vestibular signals. Based on a marginalization technique, population decoding reveals that heading and object direction can be more effectively dissociated from MSTd responses than MSTl responses. Our findings help to clarify the respective contributions that MSTl and MSTd make to processing of object motion and self-motion, although our conclusions may be somewhat specific to the multipart moving objects that we employed. NEW & NOTEWORTHY Retinal image motion reflects contributions from both the observer's self-motion and the movement of objects in the environment. The neural mechanisms by which the brain dissociates self-motion and object motion remain unclear. This study provides the first systematic examination of how the lateral subdivision of area MST (MSTl) contributes to dissociating object motion and self-motion. We also examine, for the first time, how MSTl neurons represent translational self-motion based on both vestibular and visual cues.

Vestibular System and Self-Motion

Detection of the state of self-motion, such as the instantaneous heading direction, the traveled trajectory and traveled distance or time, is critical for efficient spatial navigation. Numerous psychophysical studies have indicated that the vestibular system, originating from the otolith and semicircular canals in our inner ears, provides robust signals for different aspects of self-motion perception. In addition, vestibular signals interact with other sensory signals such as visual optic flow to facilitate natural navigation. These behavioral results are consistent with recent findings in neurophysiological studies. In particular, vestibular activity in response to the translation or rotation of the head/body in darkness is revealed in a growing number of cortical regions, many of which are also sensitive to visual motion stimuli. The temporal dynamics of the vestibular activity in the central nervous system can vary widely, ranging from acceleration-dominant to velocity-dominant. Different temporal dynamic signals may be decoded by higher level areas for different functions. For example, the acceleration signals during the translation of body in the horizontal plane may be used by the brain to estimate the heading directions. Although translation and rotation signals arise from independent peripheral organs, that is, otolith and canals, respectively, they frequently converge onto single neurons in the central nervous system including both the brainstem and the cerebral cortex. The convergent neurons typically exhibit stronger responses during a combined curved motion trajectory which may serve as the neural correlate for complex path perception. During spatial navigation, traveled distance or time may be encoded by different population of neurons in multiple regions including hippocampal-entorhinal system, posterior parietal cortex, or frontal cortex.

Probing Sensory Readout via Combined Choice-Correlation Measures and Microstimulation Perturbation

It is controversial whether covariation between neuronal activity and perceptual choice (i.e., choice correlation) reflects the functional readout of sensory signals. Here, we combined choice-correlation measures and electrical microstimulation on a site-to-site basis in the medial superior temporal area (MST), middle temporal area (MT), and ventral intraparietal area (VIP) when macaques discriminated between motion directions in both fine and coarse tasks. Microstimulation generated comparable effects between tasks but heterogeneous effects across and within brain regions. Within the MST and MT, microstimulation significantly biased an animal's choice toward the sensory preference instead of choice-related signals of the stimulated units. This was particularly evident for sites with conflict preference of sensory and choice-related signals. In the VIP, microstimulation failed to produce significant effects in either task despite strong choice correlations presented in this area. Our results suggest that sensory readout may not be inferred from choice-related signals during perceptual decision-making tasks.

Auditory motion does not modulate spiking activity in the middle temporal and medial superior temporal visual areas

The integration of multiple sensory modalities is a key aspect of brain function, allowing animals to take advantage of concurrent sources of information to make more accurate perceptual judgments. For many years, multisensory integration in the cerebral cortex was deemed to occur only in high-level "polysensory" association areas. However, more recent studies have suggested that cross-modal stimulation can also influence neural activity in areas traditionally considered to be unimodal. In particular, several human neuroimaging studies have reported that extrastriate areas involved in visual motion perception are also activated by auditory motion, and may integrate audiovisual motion cues. However, the exact nature and extent of the effects of auditory motion on the visual cortex have not been studied at the single neuron level. We recorded the spiking activity of neurons in the middle temporal (MT) and medial superior temporal (MST) areas of anesthetized marmoset monkeys upon presentation of unimodal stimuli (moving auditory or visual patterns), as well as bimodal stimuli (concurrent audiovisual motion). Despite robust, direction selective responses to visual motion, none of the sampled neurons responded to auditory motion stimuli. Moreover, concurrent moving auditory stimuli had no significant effect on the ability of single MT and MST neurons, or populations of simultaneously recorded neurons, to discriminate the direction of motion of visual stimuli (moving random dot patterns with varying levels of motion noise). Our findings do not support the hypothesis that direct interactions between MT, MST and areas low in the hierarchy of auditory areas underlie audiovisual motion integration.

Causal Evidence of Motion Signals in Macaque Middle Temporal Area Weighted-Pooled for Global Heading Perception

Accurate heading perception relies on visual information integrated across a wide field, that is, optic flow. Numerous computational studies have speculated how local visual information might be pooled by the brain to compute heading, but these hypotheses lack direct neurophysiological support. In the current study, we instructed human and monkey subjects to judge heading directions based on global optic flow. We showed that a local perturbation cue applied within only a small part of the visual field could bias the subjects' heading judgments, and shift the neuronal tuning in the macaque middle temporal (MT) area at the same time. Electrical microstimulation in MT significantly biased the animals' heading judgments predictable from the tuning of the stimulated neurons. Masking the visual stimuli within these neurons' receptive fields could not remove the stimulation effect, indicating a sufficient role of the MT signals pooled by downstream neurons for global heading estimation. Interestingly, this pooling is not homogeneous because stimulating neurons with excitatory surrounds produced relatively larger effects than stimulating neurons with inhibitory surrounds. Thus our data not only provide direct causal evidence, but also new insights into the neural mechanisms of pooling local motion information for global heading estimation.

A Neural Signature of Divisive Normalization at the Level of Multisensory Integration in Primate Cortex

Studies of multisensory integration by single neurons have traditionally emphasized empirical principles that describe nonlinear interactions between inputs from two sensory modalities. We previously proposed that many of these empirical principles could be explained by a divisive normalization mechanism operating in brain regions where multisensory integration occurs. This normalization model makes a critical diagnostic prediction: a non-preferred sensory input from one modality, which activates the neuron on its own, should suppress the response to a preferred input from another modality. We tested this prediction by recording from neurons in macaque area MSTd that integrate visual and vestibular cues regarding self-motion. We show that many MSTd neurons exhibit the diagnostic form of cross-modal suppression, whereas unisensory neurons in area MT do not. The normalization model also fits population responses better than a model based on subtractive inhibition. These findings provide strong support for a divisive normalization mechanism in multisensory integration.

Distributed Visual–Vestibular Processing in the Cerebral Cortex of Man and Macaque

Recent advances in understanding the neurobiological underpinnings of visual–vestibular interactions underlying self-motion perception are reviewed with an emphasis on comparisons between the macaque and human brains. In both species, several distinct cortical regions have been identified that are active during both visual and vestibular stimulation and in some of these there is clear evidence for sensory integration. Several possible cross-species homologies between cortical regions are identified. A key feature of cortical organization is that the same information is apparently represented in multiple, anatomically diverse cortical regions, suggesting that information about self-motion is used for different purposes in different brain regions.

Transformation of spatiotemporal dynamics in the macaque vestibular system from otolith afferents to cortex

Sensory signals undergo substantial recoding when neural activity is relayed from sensors through pre-thalamic and thalamic nuclei to cortex. To explore how temporal dynamics and directional tuning are sculpted in hierarchical vestibular circuits, we compared responses of macaque otolith afferents with neurons in the vestibular and cerebellar nuclei, as well as five cortical areas, to identical three-dimensional translational motion. We demonstrate a remarkable spatio-temporal transformation: otolith afferents carry spatially aligned cosine-tuned translational acceleration and jerk signals. In contrast, brainstem and cerebellar neurons exhibit non-linear, mixed selectivity for translational velocity, acceleration, jerk and position. Furthermore, these components often show dissimilar spatial tuning. Moderate further transformation of translation signals occurs in the cortex, such that similar spatio-temporal properties are found in multiple cortical areas. These results suggest that the first synapse represents a key processing element in vestibular pathways, robustly shaping how self-motion is represented in central vestibular circuits and cortical areas.

Multisensory Convergence of Visual and Vestibular Heading Cues in the Pursuit Area of the Frontal Eye Field

Both visual and vestibular sensory cues are important for perceiving one's direction of heading during self-motion. Previous studies have identified multisensory, heading-selective neurons in the dorsal medial superior temporal area (MSTd) and the ventral intraparietal area (VIP). Both MSTd and VIP have strong recurrent connections with the pursuit area of the frontal eye field (FEFsem), but whether FEFsem neurons may contribute to multisensory heading perception remain unknown. We characterized the tuning of macaque FEFsem neurons to visual, vestibular, and multisensory heading stimuli. About two-thirds of FEFsem neurons exhibited significant heading selectivity based on either vestibular or visual stimulation. These multisensory neurons shared many properties, including distributions of tuning strength and heading preferences, with MSTd and VIP neurons. Fisher information analysis also revealed that the average FEFsem neuron was almost as sensitive as MSTd or VIP cells. Visual and vestibular heading preferences in FEFsem tended to be either matched (congruent cells) or discrepant (opposite cells), such that combined stimulation strengthened heading selectivity for congruent cells but weakened heading selectivity for opposite cells. These findings demonstrate that, in addition to oculomotor functions, FEFsem neurons also exhibit properties that may allow them to contribute to a cortical network that processes multisensory heading cues.

Evidence for a Causal Contribution of Macaque Vestibular, But Not Intraparietal, Cortex to Heading Perception

Multisensory convergence of visual and vestibular signals has been observed within a network of cortical areas involved in representing heading. Vestibular-dominant heading tuning has been found in the macaque parietoinsular vestibular cortex (PIVC) and the adjacent visual posterior sylvian (VPS) area, whereas relatively balanced visual/vestibular tuning was encountered in the ventral intraparietal (VIP) area and visual-dominant tuning was found in the dorsal medial superior temporal (MSTd) area. Although the respective functional roles of these areas remain unclear, perceptual deficits in heading discrimination following reversible chemical inactivation of area MSTd area suggested that areas with vestibular-dominant heading tuning also contribute to behavior. To explore the roles of other areas in heading perception, muscimol injections were used to reversibly inactivate either the PIVC or the VIP area bilaterally in macaques. Inactivation of the anterior PIVC increased psychophysical thresholds when heading judgments were based on either optic flow or vestibular cues, although effects were stronger for vestibular stimuli. All behavioral deficits recovered within 36 h. Visual deficits were larger following inactivation of the posterior portion of the PIVC, likely because these injections encroached upon the VPS area, which contains neurons with optic flow tuning (unlike the PIVC). In contrast, VIP inactivation led to no behavioral deficits, despite the fact that VIP neurons show much stronger choice-related activity than MSTd neurons. These results suggest that the VIP area either provides a parallel and partially redundant pathway for this task, or does not participate in heading discrimination. In contrast, the PIVC/VPS area, along with the MSTd area, make causal contributions to heading perception based on either vestibular or visual signals.

SIGNIFICANCE STATEMENT

Multisensory vestibular and visual signals are found in multiple cortical areas, but their causal contribution to self-motion perception has been previously tested only in the dorsal medial superior temporal (MSTd) area. In these experiments, we show that inactivation of the parietoinsular vestibular cortex (PIVC) also results in causal deficits during heading discrimination for both visual and vestibular cues. In contrast, ventral intraparietal (VIP) area inactivation led to no behavioral deficits, despite the fact that VIP neurons show much stronger choice-related activity than MSTd or PIVC neurons. These results demonstrate that choice-related activity does not always imply a causal role in sensory perception.

Heading Tuning in Macaque Area V6

Cortical areas, such as the dorsal subdivision of the medial superior temporal area (MSTd) and the ventral intraparietal area (VIP), have been shown to integrate visual and vestibular self-motion signals. Area V6 is interconnected with areas MSTd and VIP, allowing for the possibility that V6 also integrates visual and vestibular self-motion cues. An alternative hypothesis in the literature is that V6 does not use these sensory signals to compute heading but instead discounts self-motion signals to represent object motion. However, the responses of V6 neurons to visual and vestibular self-motion cues have never been studied, thus leaving the functional roles of V6 unclear. We used a virtual reality system to examine the 3D heading tuning of macaque V6 neurons in response to optic flow and inertial motion stimuli. We found that the majority of V6 neurons are selective for heading defined by optic flow. However, unlike areas MSTd and VIP, V6 neurons are almost universally unresponsive to inertial motion in the absence of optic flow. We also explored the spatial reference frames of heading signals in V6 by measuring heading tuning for different eye positions, and we found that the visual heading tuning of most V6 cells was eye-centered. Similar to areas MSTd and VIP, the population of V6 neurons was best able to discriminate small variations in heading around forward and backward headings. Our findings support the idea that V6 is involved primarily in processing visual motion signals and does not appear to play a role in visual-vestibular integration for self-motion perception.

SIGNIFICANCE STATEMENT

To understand how we successfully navigate our world, it is important to understand which parts of the brain process cues used to perceive our direction of self-motion (i.e., heading). Cortical area V6 has been implicated in heading computations based on human neuroimaging data, but direct measurements of heading selectivity in individual V6 neurons have been lacking. We provide the first demonstration that V6 neurons carry 3D visual heading signals, which are represented in an eye-centered reference frame. In contrast, we found almost no evidence for vestibular heading signals in V6, indicating that V6 is unlikely to contribute to multisensory integration of heading signals, unlike other cortical areas. These findings provide important constraints on the roles of V6 in self-motion perception.

Human visual and vestibular heading perception in the vertical planes

Heading estimation has not previously been reported in the vertical planes. This is a potentially interesting issue because although distribution of neuronal direction sensitivities is near uniform for vertical headings, there is an overrepresentation of otolith organs sensitive to motion in the horizontal relative to the vertical plane. Furthermore, thresholds of horizontal motion perception are considerably lower than those of vertical motion which has the potential to bias heading perception. The current data from 14 human subjects (age 19 to 67) measured heading estimation in response to vestibular motion of 14 cm (28 cm/s) over a 360° of headings at 5° intervals. An analogous visual motion was tested in separate trials. In this study, earth and head vertical/horizontal were always aligned. Results demonstrated that the horizontal component of heading was overestimated relative to the vertical component for vestibular heading stimuli in the coronal (skew) and sagittal (elevation) planes. For visual headings, the bias was much smaller and in the opposite direction such that the vertical component of heading was overestimated. Subjects older than 50 had significantly worse precision and larger biases relative to that of younger subjects for the vestibular conditions, although visual heading estimates were similar. A vector addition model was fit to the data which explains the observed heading biases by the known distribution of otolith organs in humans. The greatly decreased precision with age is explained by the model with decreases in end organ numbers, and relatively greater loss of otoliths that are sensitive to vertical motion.

Vestibular activation differentially modulates human early visual cortex and V5/MT excitability and response entropy

Head movement imposes the additional burdens on the visual system of maintaining visual acuity and determining the origin of retinal image motion (i.e., self-motion vs. object-motion). Although maintaining visual acuity during self-motion is effected by minimizing retinal slip via the brainstem vestibular-ocular reflex, higher order visuovestibular mechanisms also contribute. Disambiguating self-motion versus object-motion also invokes higher order mechanisms, and a cortical visuovestibular reciprocal antagonism is propounded. Hence, one prediction is of a vestibular modulation of visual cortical excitability and indirect measures have variously suggested none, focal or global effects of activation or suppression in human visual cortex. Using transcranial magnetic stimulation-induced phosphenes to probe cortical excitability, we observed decreased V5/MT excitability versus increased early visual cortex (EVC) excitability, during vestibular activation. In order to exclude nonspecific effects (e.g., arousal) on cortical excitability, response specificity was assessed using information theory, specifically response entropy. Vestibular activation significantly modulated phosphene response entropy for V5/MT but not EVC, implying a specific vestibular effect on V5/MT responses. This is the first demonstration that vestibular activation modulates human visual cortex excitability. Furthermore, using information theory, not previously used in phosphene response analysis, we could distinguish between a specific vestibular modulation of V5/MT excitability from a nonspecific effect at EVC.

Direction specific biases in human visual and vestibular heading perception

Heading direction is determined from visual and vestibular cues. Both sensory modalities have been shown to have better direction discrimination for headings near straight ahead. Previous studies of visual heading estimation have not used the full range of stimuli, and vestibular heading estimation has not previously been reported. The current experiments measure human heading estimation in the horizontal plane to vestibular, visual, and spoken stimuli. The vestibular and visual tasks involved 16 cm of platform or visual motion. The spoken stimulus was a voice command speaking a heading angle. All conditions demonstrated direction dependent biases in perceived headings such that biases increased with headings further from the fore-aft axis. The bias was larger with the visual stimulus when compared with the vestibular stimulus in all 10 subjects. For the visual and vestibular tasks precision was best for headings near fore-aft. The spoken headings had the least bias, and the variation in precision was less dependent on direction. In a separate experiment when headings were limited to ±45°, the biases were much less, demonstrating the range of headings influences perception. There was a strong and highly significant correlation between the bias curves for visual and spoken stimuli in every subject. The correlation between visual-vestibular and vestibular-spoken biases were weaker but remained significant. The observed biases in both visual and vestibular heading perception qualitatively resembled predictions of a recent population vector decoder model (Gu et al., 2010) based on the known distribution of neuronal sensitivities.

Causal links between dorsal medial superior temporal area neurons and multisensory heading perception

The dorsal medial superior temporal area (MSTd) in the extrastriate visual cortex is thought to play an important role in heading perception because neurons in this area are tuned to both optic flow and vestibular signals. MSTd neurons also show significant correlations with perceptual judgments during a fine heading direction discrimination task. To test for a causal link with heading perception, we used microstimulation and reversible inactivation techniques to artificially perturb MSTd activity while monitoring behavioral performance. Electrical microstimulation significantly biased monkeys' heading percepts based on optic flow, but did not significantly impact vestibular heading judgments. The latter result may be due to the fact that vestibular heading preferences in MSTd are more weakly clustered than visual preferences and multiunit tuning for vestibular stimuli is weak. Reversible chemical inactivation, however, increased behavioral thresholds when heading judgments were based on either optic flow or vestibular cues, although the magnitude of the effects was substantially stronger for optic flow. Behavioral deficits in a combined visual/vestibular stimulus condition were intermediate between the single-cue effects. Despite deficits in discrimination thresholds, animals were able to combine visual and vestibular cues near optimally, even after large bilateral muscimol injections into MSTd. Simulations show that the overall pattern of results following inactivation is consistent with a mixture of contributions from MSTd and other areas with vestibular-dominant tuning for heading. Our results support a causal link between MSTd neurons and multisensory heading perception but suggest that other multisensory brain areas also contribute.

Estimating distance during self-motion: a role for visual-vestibular interactions

A fundamental challenge for the visual system is to extract the 3D spatial structure of the environment. When an observer translates without moving the eyes, the retinal speed of a stationary object is related to its distance by a scale factor that depends on the velocity of the observer's self-motion. Here, we aim to test whether the brain uses vestibular cues to self-motion to estimate distance to stationary surfaces in the environment. This relationship was systematically probed using a two-alternative forced-choice task in which distance perceived from monocular image motion during passive body translation was compared to distance perceived from binocular disparity while subjects were stationary. We show that perceived distance from motion depended on both observer velocity and retinal speed. For a given head speed, slower retinal speeds led to the perception of farther distances. Likewise, for a given retinal speed, slower head speeds led to the perception of nearer distances. However, these relationships were weak in some subjects and absent in others, and distance estimated from self-motion and retinal image motion was substantially compressed relative to distance estimated from binocular disparity. Overall, our findings suggest that the combination of retinal image motion and vestibular signals related to head velocity can provide a rudimentary capacity for distance estimation.

Representation of vestibular and visual cues to self-motion in ventral intraparietal cortex

Convergence of vestibular and visual motion information is important for self-motion perception. One cortical area that combines vestibular and optic flow signals is the ventral intraparietal area (VIP). We characterized unisensory and multisensory responses of macaque VIP neurons to translations and rotations in three dimensions. Approximately one-half of VIP cells show significant directional selectivity in response to optic flow, one-half show tuning to vestibular stimuli, and one-third show multisensory responses. Visual and vestibular direction preferences of multisensory VIP neurons could be congruent or opposite. When visual and vestibular stimuli were combined, VIP responses could be dominated by either input, unlike the medial superior temporal area (MSTd) where optic flow tuning typically dominates or the visual posterior sylvian area (VPS) where vestibular tuning dominates. Optic flow selectivity in VIP was weaker than in MSTd but stronger than in VPS. In contrast, vestibular tuning for translation was strongest in VPS, intermediate in VIP, and weakest in MSTd. To characterize response dynamics, direction-time data were fit with a spatiotemporal model in which temporal responses were modeled as weighted sums of velocity, acceleration, and position components. Vestibular responses in VIP reflected balanced contributions of velocity and acceleration, whereas visual responses were dominated by velocity. Timing of vestibular responses in VIP was significantly faster than in MSTd, whereas timing of optic flow responses did not differ significantly among areas. These findings suggest that VIP may be proximal to MSTd in terms of vestibular processing but hierarchically similar to MSTd in terms of optic flow processing.

Vestibular inputs to human motion-sensitive visual cortex

Two crucial sources of information available to an organism when moving through an environment are visual and vestibular stimuli. Macaque cortical area MSTd processes visual motion, including cues to self-motion arising from optic flow and also receives information about self-motion from the vestibular system. In humans, whether human MST (hMST) receives vestibular afferents is unknown. We have combined 2 techniques, galvanic vestibular stimulation and functional MRI (fMRI), to show that hMST is strongly activated by vestibular stimulation in darkness, whereas adjacent area MT is unaffected. The activity cannot be explained in terms of somatosensory stimulation at the electrode site. Vestibular input appears to be confined to the anterior portion of hMST, suggesting that hMST as conventionally defined may contain 2 subregions. Vestibular activity was also seen in another area previously implicated in processing visual cues to self-motion, namely the cingulate sulcus visual area (CSv), but not in visual area V6. The results suggest that cross-modal convergence of cues to self-motion occurs in both hMST and CSv.

Visual motion responses in the posterior cingulate sulcus: a comparison to V5/MT and MST

Motion processing regions apart from V5+/MT+ are still relatively poorly understood. Here, we used functional magnetic resonance imaging to perform a detailed functional analysis of the recently described cingulate sulcus visual area (CSv) in the dorsal posterior cingulate cortex. We used distinct types of visual motion stimuli to compare CSv with V5/MT and MST, including a visual pursuit paradigm. Both V5/MT and MST preferred 3D flow over 2D planar motion, responded less yet substantially to random motion, had a strong preference for contralateral versus ipsilateral stimulation, and responded nearly equally to contralateral and to full-field stimuli. In contrast, CSv had a pronounced preference to 2D planar motion over 3D flow, did not respond to random motion, had a weak and nonsignificant lateralization that was significantly smaller than that of MST, and strongly preferred full-field over contralateral stimuli. In addition, CSv had a better capability to integrate eye movements with retinal motion compared with V5/MT and MST. CSv thus differs from V5+/MT+ by its unique preference to full-field, coherent, and planar motion cues. These results place CSv in a good position to process visual cues related to self-induced motion, in particular those associated to eye or lateral head movements.

A comparison of vestibular spatiotemporal tuning in macaque parietoinsular vestibular cortex, ventral intraparietal area, and medial superior temporal area

Vestibular responses have been reported in the parietoinsular vestibular cortex (PIVC), the ventral intraparietal area (VIP), and the dorsal medial superior temporal area (MSTd) of macaques. However, differences between areas remain largely unknown, and it is not clear whether there is a hierarchy in cortical vestibular processing. We examine the spatiotemporal characteristics of macaque vestibular responses to translational motion stimuli using both empirical and model-based analyses. Temporal dynamics of direction selectivity were similar across areas, although there was a gradual shift in the time of peak directional tuning, with responses in MSTd typically being delayed by 100-150 ms relative to responses in PIVC (VIP was intermediate). Responses as a function of both stimulus direction and time were fit with a spatiotemporal model consisting of separable spatial and temporal response profiles. Temporal responses were characterized by a Gaussian function of velocity, a weighted sum of velocity and acceleration, or a weighted sum of velocity, acceleration, and position. Velocity and acceleration components contributed most to response dynamics, with a gradual shift from acceleration dominance in PIVC to velocity dominance in MSTd. The position component contributed little to temporal responses overall, but was substantially larger in MSTd than PIVC or VIP. The overall temporal delay in model fits also increased substantially from PIVC to VIP to MSTd. This gradual transformation of temporal responses suggests a hierarchy in cortical vestibular processing, with PIVC being most proximal to the vestibular periphery and MSTd being most distal.

Visuomotor Velocity Transformations for Smooth Pursuit Eye Movements

Smooth pursuit eye movements are driven by retinal motion signals. These retinal motion signals are converted into motor commands that obey Listing's law (i.e., no accumulation of ocular torsion). The fact that smooth pursuit follows Listing's law is often taken as evidence that no explicit reference frame transformation between the retinal velocity input and the head-centered motor command is required. Such eye-position-dependent reference frame transformations between eye- and head-centered coordinates have been well-described for saccades to static targets. Here we suggest that such an eye (and head)-position-dependent reference frame transformation is also required for target motion (i.e., velocity) driving smooth pursuit eye movements. Therefore we tested smooth pursuit initiation under different three-dimensional eye positions and compared human performance to model simulations. We specifically tested if the ocular rotation axis changed with vertical eye position, if the misalignment of the spatial and retinal axes during oblique fixations was taken into account, and if ocular torsion (due to head roll) was compensated for. If no eye-position-dependent velocity transformation was used, the pursuit initiation should follow the retinal direction, independently of eye position; in contrast, a correct visuomotor velocity transformation would result in spatially correct pursuit initiation. Overall subjects accounted for all three components of the visuomotor velocity transformation, but we did observe differences in the compensatory gains between individual subjects. We concluded that the brain does perform a visuomotor velocity transformation but that this transformation was prone to noise and inaccuracies of the internal model.

Decoding of MSTd population activity accounts for variations in the precision of heading perception

Humans and monkeys use both vestibular and visual motion (optic flow) cues to discriminate their direction of self-motion during navigation. A striking property of heading perception from optic flow is that discrimination is most precise when subjects judge small variations in heading around straight ahead, whereas thresholds rise precipitously when subjects judge heading around an eccentric reference. We show that vestibular heading discrimination thresholds in both humans and macaques also show a consistent, but modest, dependence on reference direction. We used computational methods (Fisher information, maximum likelihood estimation, and population vector decoding) to show that population activity in area MSTd predicts the dependence of heading thresholds on reference eccentricity. This dependence arises because the tuning functions for most neurons have a steep slope for directions near straight forward. Our findings support the notion that population activity in extrastriate cortex limits the precision of both visual and vestibular heading perception.

Responses of ventral posterior thalamus neurons to three-dimensional vestibular and optic flow stimulation

Multisensory neurons tuned to both vestibular and visual motion (optic flow) signals are found in several cortical areas in the dorsal visual stream. Here we examine whether such convergence occurs subcortically in the macaque thalamus. We searched the ventral posterior nuclei, including the anterior pulvinar, as well as the ventro-lateral and ventral posterior lateral nuclei, areas that receive vestibular signals from brain stem and deep cerebellar nuclei. Approximately a quarter of cells responded to three-dimensional (3D) translational and/or rotational motion. More than half of the responsive cells were convergent, thus responded during both rotation and translation. The preferred axes of translation/rotation were distributed throughout 3D space. The majority of the neurons were excited, but some were inhibited, during rotation/translation in darkness. Only a couple of neurons were multisensory being tuned to both vestibular and optic flow stimuli. We conclude that multisensory vestibular/optic flow neurons, which are commonly found in cortical visual and visuomotor areas, are rare in the ventral posterior thalamus.

MT neurons combine visual motion with a smooth eye movement signal to code depth-sign from motion parallax

The capacity to perceive depth is critical for an observer to interact with his or her surroundings. During observer movement, information about depth can be extracted from the resulting patterns of image motion on the retina (motion parallax). Without extraretinal signals related to observer movement, however, depth-sign (near versus far) from motion parallax can be ambiguous. We previously demonstrated that MT neurons combine visual motion with extraretinal signals to code depth-sign from motion parallax in the absence of other depth cues. In that study, head translations were always accompanied by compensatory tracking eye movements, allowing at least two potential sources of extraretinal input. We now show that smooth eye movement signals provide the critical extraretinal input to MT neurons for computing depth-sign from motion parallax. Our findings demonstrate a powerful modulation of MT activity by eye movements, as predicted by human studies of depth perception from motion parallax.