Reliability-Based Weighting of Visual and Vestibular Cues in Displacement Estimation

Active self-motion control and the role of agency under ambiguity

Purpose

Self-motion perception is a key factor in daily behaviours such as driving a car or piloting an aircraft. It is mainly based on visuo-vestibular integration, whose weighting mechanisms are modulated by the reliability properties of sensory inputs. Recently, it has been shown that the internal state of the operator can also modulate multisensory integration and may sharpen the representation of relevant inputs. In line with the concept of agency, it thus appears relevant to evaluate the impact of being in control of our own action on self-motion perception.

Methodology

Here, we tested two conditions of motion control (active/manual trigger versus passive/ observer condition), asking participants to discriminate between two consecutive longitudinal movements by identifying the larger displacement (displacement of higher intensity). We also tested motion discrimination under two levels of ambiguity by applying acceleration ratios that differed from our two "standard" displacements (i.e., 3 s; 0.012 m.s^-2 and 0.030 m.s^-2).

Results

We found an effect of control condition, but not of the level of ambiguity on the way participants perceived the standard displacement, i.e., perceptual bias (Point of Subjective Equality; PSE). Also, we found a significant effect of interaction between the active condition and the level of ambiguity on the ability to discriminate between displacements, i.e., sensitivity (Just Noticeable Difference; JND).

Originality

Being in control of our own motion through a manual intentional trigger of self-displacement maintains overall motion sensitivity when ambiguity increases.

Enhancement of visual cues to self-motion during a visual/vestibular conflict

Perceiving our orientation and motion requires sensory information provided by vision, our body and acceleration. Normally, these cues are redundant however in some situations they can conflict. Here, we created a visual-vestibular conflict by simulating a body-upright virtual world while participants were either standing (no conflict), supine or prone (conflict) and assessed the perception of "forward" distance travelled induced by visual motion. Some participants felt they were standing upright even when lying, indicating a visual reorientation illusion (VRI). We previously showed that when experiencing a VRI, visually induced self-motion is enhanced. Here, we determined if there was a relationship between VRI vulnerability and sensory weighting. Confirming our previous findings, the VRI-vulnerable group showed enhanced self-motion perception. We then assessed the relative weightings of visual and non-visual cues in VRI-vulnerable and VRI-resistant individuals using the Oriented Character Recognition Test. Surprisingly, VRI-vulnerable individuals weighted visual cues less and gravity cues more compared to VRI-resistant individuals. These findings are in line with robust integration where, when the difference between two cues is large, the discrepant cue (here gravity) is ignored. Ignoring the gravity cue then leads to relatively more emphasis being placed on visual information and thus a higher gain.

Predictive steering: integration of artificial motor signals in self-motion estimation

The brain's computations for active and passive self-motion estimation can be unified with a single model that optimally combines vestibular and visual signals with sensory predictions based on efference copies. It is unknown whether this theoretical framework also applies to the integration of artificial motor signals, such as those that occur when driving a car, or whether self-motion estimation in this situation relies on sole feedback control. Here, we examined if training humans to control a self-motion platform leads to the construction of an accurate internal model of the mapping between the steering movement and the vestibular reafference. Participants (n = 15) sat on a linear motion platform and actively controlled the platform's velocity using a steering wheel to translate their body to a memorized visual target (motion condition). We compared their steering behavior to that of participants (n = 15) who remained stationary and instead aligned a nonvisible line with the target (stationary condition). To probe learning, the gain between the steering wheel angle and the platform or line velocity changed abruptly twice during the experiment. These gain changes were virtually undetectable in the displacement error in the motion condition, whereas clear deviations were observed in the stationary condition, showing that participants in the motion condition made within-trial changes to their steering behavior. We conclude that vestibular feedback allows not only the online control of steering but also a rapid adaptation to the gain changes to update the brain's internal model of the mapping between the steering movement and the vestibular reafference.NEW & NOTEWORTHY Perception of self-motion is known to depend on the integration of sensory signals and, when the motion is self-generated, the predicted sensory reafference based on motor efference copies. Here we show, using a closed-loop steering experiment with a direct coupling between the steering movement and the vestibular self-motion feedback, that humans are also able to integrate artificial motor signals, like the motor signals that occur when driving a car.

Sensory Evidence Accumulation Using Optic Flow in a Naturalistic Navigation Task

Sensory evidence accumulation is considered a hallmark of decision-making in noisy environments. Integration of sensory inputs has been traditionally studied using passive stimuli, segregating perception from action. Lessons learned from this approach, however, may not generalize to ethological behaviors like navigation, where there is an active interplay between perception and action. We designed a sensory-based sequential decision task in virtual reality in which humans and monkeys navigated to a memorized location by integrating optic flow generated by their own joystick movements. A major challenge in such closed-loop tasks is that subjects' actions will determine future sensory input, causing ambiguity about whether they rely on sensory input rather than expectations based solely on a learned model of the dynamics. To test whether subjects integrated optic flow over time, we used three independent experimental manipulations, unpredictable optic flow perturbations, which pushed subjects off their trajectory; gain manipulation of the joystick controller, which changed the consequences of actions; and manipulation of the optic flow density, which changed the information borne by sensory evidence. Our results suggest that both macaques (male) and humans (female/male) relied heavily on optic flow, thereby demonstrating a critical role for sensory evidence accumulation during naturalistic action-perception closed-loop tasks.SIGNIFICANCE STATEMENT The temporal integration of evidence is a fundamental component of mammalian intelligence. Yet, it has traditionally been studied using experimental paradigms that fail to capture the closed-loop interaction between actions and sensations inherent in real-world continuous behaviors. These conventional paradigms use binary decision tasks and passive stimuli with statistics that remain stationary over time. Instead, we developed a naturalistic visuomotor visual navigation paradigm that mimics the causal structure of real-world sensorimotor interactions and probed the extent to which participants integrate sensory evidence by adding task manipulations that reveal complementary aspects of the computation.

Influence of sensory modality and control dynamics on human path integration

Path integration is a sensorimotor computation that can be used to infer latent dynamical states by integrating self-motion cues. We studied the influence of sensory observation (visual/vestibular) and latent control dynamics (velocity/acceleration) on human path integration using a novel motion-cueing algorithm. Sensory modality and control dynamics were both varied randomly across trials, as participants controlled a joystick to steer to a memorized target location in virtual reality. Visual and vestibular steering cues allowed comparable accuracies only when participants controlled their acceleration, suggesting that vestibular signals, on their own, fail to support accurate path integration in the absence of sustained acceleration. Nevertheless, performance in all conditions reflected a failure to fully adapt to changes in the underlying control dynamics, a result that was well explained by a bias in the dynamics estimation. This work demonstrates how an incorrect internal model of control dynamics affects navigation in volatile environments in spite of continuous sensory feedback.

The Impact of Optical Illusions on the Vestibular System

Background and Objectives

Balance control is maintained in stationary and dynamic conditions, with coordinated muscle responses generated by somatosensory, vestibular, and visual inputs. This study aimed to investigate how the vestibular system is affected in the presence of an optical illusion to better understand the interconnected pathways of the visual and vestibular systems.

Subjects and Methods

The study involved 54 young adults (27 males and 27 females) aged 18-25 years. The recruited participants were subjected to the cervical vestibular evoked myogenic potentials (cVEMP) test and video head impulse test (vHIT). The cVEMP and vHIT tests were performed once each in the absence and presence of an optical illusion. In addition, after each test, whether the individuals felt balanced was determined using a questionnaire.

Results

cVEMP results in the presence of the optical illusion showed shortened latencies and increased amplitudes for the left side in comparison to the results in the absence of the optical illusion (p≤0.05). When vHIT results were compared, it was seen that the right lateral and bilateral anterior canal gains were increased, almost to 1.0 (p<0.05).

Conclusions

It is thought that when the visual-vestibular inputs are incompatible with each other, the sensory reweighting mechanism is activated, and this mechanism strengthens the more reliable (vestibular) inputs, while suppressing the less reliable (visual) inputs. As long as the incompatible condition persists, the sensory reweighting mechanism will continue to operate, thanks to the feedback loop from the efferent vestibular system.

Perceptual-motor styles

Even for a stereotyped task, sensorimotor behavior is generally variable due to noise, redundancy, adaptability, learning or plasticity. The sources and significance of different kinds of behavioral variability have attracted considerable attention in recent years. However, the idea that part of this variability depends on unique individual strategies has been explored to a lesser extent. In particular, the notion of style recurs infrequently in the literature on sensorimotor behavior. In general use, style refers to a distinctive manner or custom of behaving oneself or of doing something, especially one that is typical of a person, group of people, place, context, or period. The application of the term to the domain of perceptual and motor phenomenology opens new perspectives on the nature of behavioral variability, perspectives that are complementary to those typically considered in the studies of sensorimotor variability. In particular, the concept of style may help toward the development of personalised physiology and medicine by providing markers of individual behaviour and response to different stimuli or treatments. Here, we cover some potential applications of the concept of perceptual-motor style to different areas of neuroscience, both in the healthy and the diseased. We prefer to be as general as possible in the types of applications we consider, even at the expense of running the risk of encompassing loosely related studies, given the relative novelty of the introduction of the term perceptual-motor style in neurosciences.

Visual-vestibular integration is preserved with healthy aging in a simple acceleration detection task

Aging is associated with a gradual decline in the sensory systems and noisier sensory information. Some research has found that older adults compensate for this with enhanced multisensory integration. However, less is known about how aging influences visual-vestibular integration, an ability that underlies self-motion perception. We examined how visual-vestibular integration changes in participants from across the lifespan (18-79 years old) with a simple reaction time task. Participants were instructed to respond to visual (optic flow) and vestibular (inertial motion) acceleration cues, presented either alone or at a stimulus onset asynchrony. We measured reaction times and computed the violation area relative to the race model inequality as a measure of visual-vestibular integration. Across all ages, the greatest visual-vestibular integration occurred when the vestibular cue was presented first. Age was associated with longer reaction times and a significantly lower detection rate in the vestibular-only condition, a finding that is consistent with an age-related increase in vestibular noise. Although the relationship between age and visual-vestibular integration was positive, the effect size was very small and did not reach statistical significance. Our results suggest that although age is associated with a significant increase in vestibular perceptual threshold, the relative amount of visual-vestibular integration remains largely intact.

When gravity is not where it should be: How perceived orientation affects visual self-motion processing

Human perception is based on expectations. We expect visual upright and gravity upright, sensed through vision, vestibular and other sensory systems, to agree. Equally, we expect that visual and vestibular information about self-motion will correspond. What happens when these assumptions are violated? Tilting a person from upright so that gravity is not where it should be impacts both visually induced self-motion (vection) and the perception of upright. How might the two be connected? Using virtual reality, we varied the strength of visual orientation cues, and hence the probability of participants experiencing a visual reorientation illusion (VRI) in which visual cues to orientation dominate gravity, using an oriented corridor and a starfield while also varying head-on-trunk orientation and body posture. The effectiveness of the optic flow in simulating self-motion was assessed by how much visual motion was required to evoke the perception that the participant had reached the position of a previously presented target. VRI was assessed by questionnaire When participants reported higher levels of VRI they also required less visual motion to evoke the sense of traveling through a given distance, regardless of head or body posture, or the type of visual environment. We conclude that experiencing a VRI, in which visual-vestibular conflict is resolved and the direction of upright is reinterpreted, affects the effectiveness of optic flow at simulating motion through the environment. Therefore, any apparent effect of head or body posture or type of environment are largely indirect effects related instead, to the level of VRI experienced by the observer. We discuss potential mechanisms for this such as reinterpreting gravity information or altering the weighting of orientation cues.

Variance based weighting of multisensory head rotation signals for verticality perception

We tested the hypothesis that the brain uses a variance-based weighting of multisensory cues to estimate head rotation to perceive which way is up. The hypothesis predicts that the known bias in perceived vertical, which occurs when the visual environment is rotated in a vertical-plane, will be reduced by the addition of visual noise. Ten healthy participants sat head-fixed in front of a vertical screen presenting an annulus filled with coloured dots, which could rotate clockwise or counter-clockwise at six angular velocities (1, 2, 4, 6, 8, 16°/s) and with six levels of noise (0, 25, 50, 60, 75, 80%). Participants were required to keep a central bar vertical by rotating a hand-held dial. Continuous adjustments of the bar were required to counteract low-amplitude low-frequency noise that was added to the bar's angular position. During visual rotation, the bias in verticality perception increased over time to reach an asymptotic value. Increases in visual rotation velocity significantly increased this bias, while the addition of visual noise significantly reduced it, but did not affect perception of visual rotation velocity. The biasing phenomena were reproduced by a model that uses a multisensory variance-weighted estimate of head rotation velocity combined with a gravito-inertial acceleration signal (GIA) from the vestibular otoliths. The time-dependent asymptotic behaviour depends on internal feedback loops that act to pull the brain's estimate of gravity direction towards the GIA signal. The model's prediction of our experimental data furthers our understanding of the neural processes underlying human verticality perception.

Multisensory Influences on Driver Steering During Curve Navigation

OBJECTIVE

The effects of inertial (vestibular and somatosensory) information on driver steering during curve navigation were investigated, using an electric four-wheel mobility vehicle outfitted with a steering wheel and a portable virtual reality system.

BACKGROUND

When driving, multiple sources of perceptual information are available. Researchers have focused on visual information, which plays a critical role in steering control. However, it is not yet well established how inertial information might contribute.

METHODS

I biased inertial cues by varying visual/inertial gains (doubled, halved, reversed), as drivers negotiated curving paths, and measured steering accuracy and efficiency. I also assessed whether being exposed to inertial biases had an impact on postbias steering by comparing pre- and posttest session performance measures.

RESULTS

Doubling or halving inertial cues had little effect on steering performance. Inertial information only disrupted steering when it was reversed with respect to visual information. Over time, the influence of this extreme inertial bias was reduced though not eliminated. Postbias curve navigation performance was not impacted, likely because participants had learned to disregard, rather than integrate, biased inertial cues.

CONCLUSION

Results suggest that biased inertial information has little influence on curve navigation performance when visual information is available.

APPLICATION

Though inertial cues may be important for open-loop steering, when visual cues are unavailable, their role in closed-loop steering seems less influential. This has implications for driving simulation and suggests that inertial discrepancies due to limitations in motion-cuing capabilities may not be all that problematic for the simulation of closed-loop curve steering tasks.

Causal inference for spatial constancy across whole body motion

The brain uses self-motion information to internally update egocentric representations of locations of remembered world-fixed visual objects. If a discrepancy is observed between this internal update and reafferent visual feedback, this could be either due to an inaccurate update or because the object has moved during the motion. To optimally infer the object's location it is therefore critical for the brain to estimate the probabilities of these two causal structures and accordingly integrate and/or segregate the internal and sensory estimates. To test this hypothesis, we designed a spatial updating task involving passive whole body translation. Participants, seated on a vestibular sled, had to remember the world-fixed position of a visual target. Immediately after the translation, the reafferent visual feedback was provided by flashing a second target around the estimated "updated" target location, and participants had to report the initial target location. We found that the participants' responses were systematically biased toward the position of the second target position for relatively small but not for large differences between the "updated" and the second target location. This pattern was better captured by a Bayesian causal inference model than by alternative models that would always either integrate or segregate the internally updated target location and the visual feedback. Our results suggest that the brain implicitly represents the posterior probability that the internally updated estimate and the visual feedback come from a common cause and uses this probability to weigh the two sources of information in mediating spatial constancy across whole body motion. NEW & NOTEWORTHY When we move, egocentric representations of object locations require internal updating to keep them in register with their true world-fixed locations. How does this mechanism interact with reafferent visual input, given that objects typically do not disappear from view? Here we show that the brain implicitly represents the probability that both types of information derive from the same object and uses this probability to weigh their contribution for achieving spatial constancy across whole body motion.

Vestibular System and Self-Motion

Detection of the state of self-motion, such as the instantaneous heading direction, the traveled trajectory and traveled distance or time, is critical for efficient spatial navigation. Numerous psychophysical studies have indicated that the vestibular system, originating from the otolith and semicircular canals in our inner ears, provides robust signals for different aspects of self-motion perception. In addition, vestibular signals interact with other sensory signals such as visual optic flow to facilitate natural navigation. These behavioral results are consistent with recent findings in neurophysiological studies. In particular, vestibular activity in response to the translation or rotation of the head/body in darkness is revealed in a growing number of cortical regions, many of which are also sensitive to visual motion stimuli. The temporal dynamics of the vestibular activity in the central nervous system can vary widely, ranging from acceleration-dominant to velocity-dominant. Different temporal dynamic signals may be decoded by higher level areas for different functions. For example, the acceleration signals during the translation of body in the horizontal plane may be used by the brain to estimate the heading directions. Although translation and rotation signals arise from independent peripheral organs, that is, otolith and canals, respectively, they frequently converge onto single neurons in the central nervous system including both the brainstem and the cerebral cortex. The convergent neurons typically exhibit stronger responses during a combined curved motion trajectory which may serve as the neural correlate for complex path perception. During spatial navigation, traveled distance or time may be encoded by different population of neurons in multiple regions including hippocampal-entorhinal system, posterior parietal cortex, or frontal cortex.

Effect of vibration during visual-inertial integration on human heading perception during eccentric gaze

Heading direction is determined from visual and inertial cues. Visual headings use retinal coordinates while inertial headings use body coordinates. Thus during eccentric gaze the same heading may be perceived differently by visual and inertial modalities. Stimulus weights depend on the relative reliability of these stimuli, but previous work suggests that the inertial heading may be given more weight than predicted. These experiments only varied the visual stimulus reliability, and it is unclear what occurs with variation in inertial reliability. Five human subjects completed a heading discrimination task using 2s of translation with a peak velocity of 16cm/s. Eye position was ±25° left/right with visual, inertial, or combined motion. The visual motion coherence was 50%. Inertial stimuli included 6 Hz vertical vibration with 0, 0.10, 0.15, or 0.20cm amplitude. Subjects reported perceived heading relative to the midline. With an inertial heading, perception was biased 3.6° towards the gaze direction. Visual headings biased perception 9.6° opposite gaze. The inertial threshold without vibration was 4.8° which increased significantly to 8.8° with vibration but the amplitude of vibration did not influence reliability. With visual-inertial headings, empirical stimulus weights were calculated from the bias and compared with the optimal weight calculated from the threshold. In 2 subjects empirical weights were near optimal while in the remaining 3 subjects the inertial stimuli were weighted greater than optimal predictions. On average the inertial stimulus was weighted greater than predicted. These results indicate multisensory integration may not be a function of stimulus reliability when inertial stimulus reliability is varied.

State Estimation for Early Feedback Responses in Reaching: Intramodal or Multimodal?

Humans are highly skilled in controlling their reaching movements, making fast and task-dependent movement corrections to unforeseen perturbations. To guide these corrections, the neural control system requires a continuous, instantaneous estimate of the current state of the arm and body in the world. According to Optimal Feedback Control theory, this estimate is multimodal and constructed based on the integration of forward motor predictions and sensory feedback, such as proprioceptive, visual and vestibular information, modulated by context, and shaped by past experience. But how can a multimodal estimate drive fast movement corrections, given that the involved sensory modalities have different processing delays, different coordinate representations, and different noise levels? We develop the hypothesis that the earliest online movement corrections are based on multiple single modality state estimates rather than one combined multimodal estimate. We review studies that have investigated online multimodal integration for reach control and offer suggestions for experiments to test for the existence of intramodal state estimates. If proven true, the framework of Optimal Feedback Control needs to be extended with a stage of intramodal state estimation, serving to drive short-latency movement corrections.

Multisensory Perception of Contradictory Information in an Environment of Varying Reliability: Evidence for Conscious Perception and Optimal Causal Inference

Two psychophysical experiments examined multisensory integration of visual-auditory (Experiment 1) and visual-tactile-auditory (Experiment 2) signals. Participants judged the location of these multimodal signals relative to a standard presented at the median plane of the body. A cue conflict was induced by presenting the visual signals with a constant spatial discrepancy to the other modalities. Extending previous studies, the reliability of certain modalities (visual in Experiment 1, visual and tactile in Experiment 2) was varied from trial to trial by presenting signals with either strong or weak location information (e.g., a relatively dense or dispersed dot cloud as visual stimulus). We investigated how participants would adapt to the cue conflict from the contradictory information under these varying reliability conditions and whether participants had insight to their performance. During the course of both experiments, participants switched from an integration strategy to a selection strategy in Experiment 1 and to a calibration strategy in Experiment 2. Simulations of various multisensory perception strategies proposed that optimal causal inference in a varying reliability environment not only depends on the amount of multimodal discrepancy, but also on the relative reliability of stimuli across the reliability conditions.