Multisensory Calibration: A Variety of Slow and Fast Brain Processes Throughout the Lifespan

From before we are born, throughout development, adulthood, and aging, we are immersed in a multisensory world. At each of these stages, our sensory cues are constantly changing, due to body, brain, and environmental changes. While integration of information from our different sensory cues improves precision, this only improves accuracy if the underlying cues are unbiased. Thus, multisensory calibration is a vital and ongoing process. To meet this grand challenge, our brains have evolved a variety of mechanisms. First, in response to a systematic discrepancy between sensory cues (without external feedback) the cues calibrate one another (unsupervised calibration). Second, multisensory function is calibrated to external feedback (supervised calibration). These two mechanisms superimpose. While the former likely reflects a lower level mechanism, the latter likely reflects a higher level cognitive mechanism. Indeed, neural correlates of supervised multisensory calibration in monkeys were found in higher level multisensory cortical area VIP, but not in the relatively lower level multisensory area MSTd. In addition, even without a cue discrepancy (e.g., when experiencing stimuli from different sensory cues in series) the brain monitors supra-modal statistics of events in the environment and adapts perception cross-modally. This too comprises a variety of mechanisms, including confirmation bias to prior choices, and lower level cross-sensory adaptation. Further research into the neuronal underpinnings of the broad and diverse functions of multisensory calibration, with improved synthesis of theories is needed to attain a more comprehensive understanding of multisensory brain function.

Multisensory decisions from self to world

Classic Bayesian models of perceptual inference describe how an ideal observer would integrate 'unisensory' measurements (multisensory integration) and attribute sensory signals to their origin(s) (causal inference). However, in the brain, sensory signals are always received in the context of a multisensory bodily state-namely, in combination with other senses. Moreover, sensory signals from both interoceptive sensing of one's own body and exteroceptive sensing of the world are highly interdependent and never occur in isolation. Thus, the observer must fundamentally determine whether each sensory observation is from an external (versus internal, self-generated) source to even be considered for integration. Critically, solving this primary causal inference problem requires knowledge of multisensory and sensorimotor dependencies. Thus, multisensory processing is needed to separate sensory signals. These multisensory processes enable us to simultaneously form a sense of self and form distinct perceptual decisions about the external world. In this opinion paper, we review and discuss the similarities and distinctions between multisensory decisions underlying the sense of self and those directed at acquiring information about the world. We call attention to the fact that heterogeneous multisensory processes take place all along the neural hierarchy (even in forming 'unisensory' observations) and argue that more integration of these aspects, in theory and experiment, is required to obtain a more comprehensive understanding of multisensory brain function. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Modality-specific sensory and decisional carryover effects in duration perception

BACKGROUND

The brain uses recent history when forming perceptual decisions. This results in carryover effects in perception. Although separate sensory and decisional carryover effects have been shown in many perceptual tasks, their existence and nature in temporal processing are unclear. Here, we investigated whether and how previous stimuli and previous choices affect subsequent duration perception, in vision and audition.

RESULTS

In a series of three experiments, participants were asked to classify visual or auditory stimuli into "shorter" or "longer" duration categories. In experiment 1, visual and auditory stimuli were presented in separate blocks. Results showed that current duration estimates were repelled away from the previous trial's stimulus duration, but attracted towards the previous choice, in both vision and audition. In experiment 2, visual and auditory stimuli were pseudorandomly presented in one block. We found that sensory and decisional carryover effects occurred only when previous and current stimuli were from the same modality. Experiment 3 further investigated the stimulus dependence of carryover effects within each modality. In this experiment, visual stimuli with different shape topologies (or auditory stimuli with different audio frequencies) were pseudorandomly presented in one visual (or auditory) block. Results demonstrated sensory carryover (within each modality) despite task-irrelevant differences in visual shape topology or audio frequency. By contrast, decisional carryover was reduced (but still present) across different visual topologies and completely absent across different audio frequencies.

CONCLUSIONS

These results suggest that serial dependence in duration perception is modality-specific. Moreover, repulsive sensory carryover effects generalize within each modality, whereas attractive decisional carryover effects are contingent on contextual details.

Contrary neuronal recalibration in different multisensory cortical areas

The adult brain demonstrates remarkable multisensory plasticity by dynamically recalibrating itself based on information from multiple sensory sources. After a systematic visual-vestibular heading offset is experienced, the unisensory perceptual estimates for subsequently presented stimuli are shifted toward each other (in opposite directions) to reduce the conflict. The neural substrate of this recalibration is unknown. Here, we recorded single-neuron activity from the dorsal medial superior temporal (MSTd), parietoinsular vestibular cortex (PIVC), and ventral intraparietal (VIP) areas in three male rhesus macaques during this visual-vestibular recalibration. Both visual and vestibular neuronal tuning curves in MSTd shifted - each according to their respective cues' perceptual shifts. Tuning of vestibular neurons in PIVC also shifted in the same direction as vestibular perceptual shifts (cells were not robustly tuned to the visual stimuli). By contrast, VIP neurons demonstrated a unique phenomenon: both vestibular and visual tuning shifted in accordance with vestibular perceptual shifts. Such that, visual tuning shifted, surprisingly, contrary to visual perceptual shifts. Therefore, while unsupervised recalibration (to reduce cue conflict) occurs in early multisensory cortices, higher-level VIP reflects only a global shift, in vestibular space.

Supervised Multisensory Calibration Signals Are Evident in VIP But Not MSTd

Multisensory plasticity enables our senses to dynamically adapt to each other and the external environment, a fundamental operation that our brain performs continuously. We searched for neural correlates of adult multisensory plasticity in the dorsal medial superior temporal area (MSTd) and the ventral intraparietal area (VIP) in 2 male rhesus macaques using a paradigm of supervised calibration. We report little plasticity in neural responses in the relatively low-level multisensory cortical area MSTd. In contrast, neural correlates of plasticity are found in higher-level multisensory VIP, an area with strong decision-related activity. Accordingly, we observed systematic shifts of VIP tuning curves, which were reflected in the choice-related component of the population response. This is the first demonstration of neuronal calibration, together with behavioral calibration, in single sessions. These results lay the foundation for understanding multisensory neural plasticity, applicable broadly to maintaining accuracy for sensorimotor tasks.SIGNIFICANCE STATEMENT Multisensory plasticity is a fundamental and continual function of the brain that enables our senses to adapt dynamically to each other and to the external environment. Yet, very little is known about the neuronal mechanisms of multisensory plasticity. In this study, we searched for neural correlates of adult multisensory plasticity in the dorsal medial superior temporal area (MSTd) and the ventral intraparietal area (VIP) using a paradigm of supervised calibration. We found little plasticity in neural responses in the relatively low-level multisensory cortical area MSTd. By contrast, neural correlates of plasticity were found in VIP, a higher-level multisensory area with strong decision-related activity. This is the first demonstration of neuronal calibration, together with behavioral calibration, in single sessions.

Increased influence of prior choices on perceptual decisions in autism

Autism spectrum disorder (ASD) manifests sensory and perceptual atypicalities. Recent theories suggest that these may reflect a reduced influence of prior information in ASD. Some studies have found reduced adaptation to recent sensory stimuli in ASD. However, the effects of prior stimuli and prior perceptual choices can counteract one-another. Here, we investigated this using two different tasks (in two different cohorts): (i) visual location discrimination and (ii) multisensory (visual-vestibular) heading discrimination. We fit the data using a logistic regression model to dissociate the specific effects of prior stimuli and prior choices. In both tasks, perceptual decisions were biased toward recent choices. Notably, the 'attractive' effect of prior choices was significantly larger in ASD (in both tasks and cohorts), while there was no difference in the influence of prior stimuli. These results challenge theories of reduced priors in ASD, and rather suggest an increased consistency bias for perceptual decisions in ASD.

Overconfidence in visual perception in parkinson's disease

Increased dependence on visual cues in Parkinson's disease (PD) can unbalance the perception-action loop, impair multisensory integration, and affect everyday function of PD patients. It is currently unknown why PD patients seem to be more reliant on their visual cues. We hypothesized that PD patients may be overconfident in the reliability (precision) of their visual cues. In this study we tested coherent visual motion perception in PD, and probed subjective (self-reported) confidence in their visual motion perception. Twenty patients with idiopathic PD, 21 healthy aged-matched controls and 20 healthy young adult participants were presented with visual stimuli of moving dots (random dot kinematograms). They were asked to report: (1) whether the aggregate motion of dots was to the left or to the right, and (2) how confident they were that their perceptual discrimination was correct. Visual motion discrimination thresholds were similar (unimpaired) in PD compared to the other groups. By contrast, PD patients were significantly overconfident in their visual perceptual decisions (p = .002 and p < .001 vs. the age-matched and young adult groups, respectively). These results suggest intact visual motion perception, but overestimation of visual cue reliability, in PD. Overconfidence in visual (vs. other, e.g., somatosensory) cues could underlie increased visual dependence and impaired multisensory/sensorimotor integration in PD. It could thereby contribute to gait and balance impairments, and affect everyday activities, such as driving. Future work should investigate and compare PD confidence in somatosensory function. A better understanding of altered sensory reliance might open up new avenues to treat debilitating PD symptoms.

Perceptual decisions are biased toward relevant prior choices

Perceptual decisions are biased by recent perceptual history-a phenomenon termed 'serial dependence.' Here, we investigated what aspects of perceptual decisions lead to serial dependence, and disambiguated the influences of low-level sensory information, prior choices and motor actions. Participants discriminated whether a brief visual stimulus lay to left/right of the screen center. Following a series of biased 'prior' location discriminations, subsequent 'test' location discriminations were biased toward the prior choices, even when these were reported via different motor actions (using different keys), and when the prior and test stimuli differed in color. By contrast, prior discriminations about an irrelevant stimulus feature (color) did not substantially influence subsequent location discriminations, even though these were reported via the same motor actions. Additionally, when color (not location) was discriminated, a bias in prior stimulus locations no longer influenced subsequent location discriminations. Although low-level stimuli and motor actions did not trigger serial-dependence on their own, similarity of these features across discriminations boosted the effect. These findings suggest that relevance across perceptual decisions is a key factor for serial dependence. Accordingly, serial dependence likely reflects a high-level mechanism by which the brain predicts and interprets new incoming sensory information in accordance with relevant prior choices.

Visual self-motion cues are impaired yet overweighted during visual-vestibular integration in Parkinson's disease

Parkinson's disease is prototypically a movement disorder. Although perceptual and motor functions are highly interdependent, much less is known about perceptual deficits in Parkinson's disease, which are less observable by nature, and might go unnoticed if not tested directly. It is therefore imperative to seek and identify these, to fully understand the challenges facing patients with Parkinson's disease. Also, perceptual deficits may be related to motor symptoms. Posture, gait and balance, affected in Parkinson's disease, rely on veridical perception of one's own motion (self-motion) in space. Yet it is not known whether self-motion perception is impaired in Parkinson's disease. Using a well-established multisensory paradigm of heading discrimination (that has not been previously applied to Parkinson's disease), we tested unisensory visual and vestibular self-motion perception, as well as multisensory integration of visual and vestibular cues, in 19 Parkinson's disease, 23 healthy age-matched and 20 healthy young-adult participants. After experiencing vestibular (on a motion platform), visual (optic flow) or multisensory (combined visual-vestibular) self-motion stimuli at various headings, participants reported whether their perceived heading was to the right or left of straight ahead. Parkinson's disease participants and age-matched controls were tested twice (Parkinson's disease participants on and off medication). Parkinson's disease participants demonstrated significantly impaired visual self-motion perception compared with age-matched controls on both visits, irrespective of medication status. Young controls performed slightly (but not significantly) better than age-matched controls and significantly better than the Parkinson's disease group. The visual self-motion perception impairment in Parkinson's disease correlated significantly with clinical disease severity. By contrast, vestibular performance was unimpaired in Parkinson's disease. Remarkably, despite impaired visual self-motion perception, Parkinson's disease participants significantly overweighted the visual cues during multisensory (visual-vestibular ) integration (compared with Bayesian predictions of optimal integration) and significantly more than controls. These findings indicate that self-motion perception in Parkinson's disease is affected by impaired visual cues and by suboptimal visual-vestibular integration (overweighting of visual cues). Notably, vestibular self-motion perception was unimpaired. Thus, visual self-motion perception is specifically impaired in early-stage Parkinson's disease. This can impact Parkinson's disease diagnosis and subtyping. Overweighting of visual cues could reflect a general multisensory integration deficit in Parkinson's disease, or specific overestimation of visual cue reliability. Finally, impaired self-motion perception in Parkinson's disease may contribute to impaired balance and gait control. Future investigation into this connection might open up new avenues of alternative therapies to better treat these difficult symptoms.

Self-motion perception in autism is compromised by visual noise but integrated optimally across multiple senses

Perceptual processing in autism spectrum disorder (ASD) is marked by superior low-level task performance and inferior complex-task performance. This observation has led to theories of defective integration in ASD of local parts into a global percept. Despite mixed experimental results, this notion maintains widespread influence and has also motivated recent theories of defective multisensory integration in ASD. Impaired ASD performance in tasks involving classic random dot visual motion stimuli, corrupted by noise as a means to manipulate task difficulty, is frequently interpreted to support this notion of global integration deficits. By manipulating task difficulty independently of visual stimulus noise, here we test the hypothesis that heightened sensitivity to noise, rather than integration deficits, may characterize ASD. We found that although perception of visual motion through a cloud of dots was unimpaired without noise, the addition of stimulus noise significantly affected adolescents with ASD, more than controls. Strikingly, individuals with ASD demonstrated intact multisensory (visual-vestibular) integration, even in the presence of noise. Additionally, when vestibular motion was paired with pure visual noise, individuals with ASD demonstrated a different strategy than controls, marked by reduced flexibility. This result could be simulated by using attenuated (less reliable) and inflexible (not experience-dependent) Bayesian priors in ASD. These findings question widespread theories of impaired global and multisensory integration in ASD. Rather, they implicate increased sensitivity to sensory noise and less use of prior knowledge in ASD, suggesting increased reliance on incoming sensory information.

Supervised calibration relies on the multisensory percept

Multisensory plasticity enables us to dynamically adapt sensory cues to one another and to the environment. Without external feedback, "unsupervised" multisensory calibration reduces cue conflict in a manner largely independent of cue reliability. But environmental feedback regarding cue accuracy ("supervised") also affects calibration. Here we measured the combined influence of cue accuracy and cue reliability on supervised multisensory calibration, using discrepant visual and vestibular motion stimuli. When the less reliable cue was inaccurate, it alone got calibrated. However, when the more reliable cue was inaccurate, cues were yoked and calibrated together in the same direction. Strikingly, the less reliable cue shifted away from external feedback, becoming less accurate. A computational model in which supervised and unsupervised calibration work in parallel, where the former only relies on the multisensory percept, but the latter can calibrate cues individually, accounts for the observed behavior. In combination, they could ultimately achieve the optimal solution of both external accuracy and internal consistency.

Microelectrode recording duration and spatial density constraints for automatic targeting of the subthalamic nucleus

BACKGROUND

Accurate detection of the boundaries of the subthalamic nucleus (STN) in deep brain stimulation (DBS) surgery using microelectrode recording (MER) is considered to refine localization and may therefore improve clinical outcome. However, MER tends to extend operation time and its cost-utility balance has been debated.

OBJECTIVES

To quantify the tradeoff between accuracy of STN localization and the spatial and temporal parameters of MER that effect the operation time using an automated detection method.

METHODS

We retrospectively estimated the accuracy of STN detection on data from 100 microelectrode trajectories. Our dense (average step = 0.12 mm) and long (average duration = 22.5 s) MER data was downsampled in the spatial and temporal domains. Then, the STN borders were detected automatically on both the downsampled and original data and compared to each other.

RESULTS

With a recording duration of 16 s, average accuracy for detecting STN entry ranged from 0.06 mm for a 0.1-mm step to 0.51 mm for a 1.0-mm step. Smaller effects were found along the temporal axis. For example, a 0.1-mm recording step yielded an STN entry average accuracy ranging from 0.06 mm for a 16-second recording duration to 0.16 mm for 0.1 s.

CONCLUSIONS

STN entry detection error was about half of the step size. Sampling duration of STN activity can be minimized to 1 s/record without compromising accuracy. We conclude that bilateral DBS surgery time utilizing MER may be significantly shortened without compromising targeting accuracy.

Levodopa and subthalamic deep brain stimulation responses are not congruent

There is a consensus that in Parkinson's disease, the extent of preoperative levodopa responsiveness predicts the efficacy of subthalamic nucleus deep brain stimulation (STN DBS). However, this may be the result of statistical methods and primary assumptions. We were able to reproduce previously published correlation results on our data (N = 49 patients). Yet, these same results were demonstrated even after random shuffling of our data. Notably, we did not observe a correlation between STN DBS efficacy and preoperative levodopa responsiveness when using their respective baselines and fractional scores of motor improvement. Furthermore, postoperative responses were not limited by preoperative scores, with tremor demonstrating the greatest discrepancy. We conclude that preoperative levodopa responsiveness does not predict or limit the outcome of STN DBS. These results imply different therapeutic mechanisms for levodopa and STN DBS and therefore question the validity of using substantial preoperative levodopa responsiveness as a selection criterion for STN DBS.

Propofol decreases neuronal population spiking activity in the subthalamic nucleus of Parkinsonian patients

BACKGROUND

Implantation of deep brain stimulation (DBS) electrodes in the subthalamic nucleus (STN) for the treatment of Parkinson disease is often performed using microelectrode recording (MER) of STN population spike activity. The extent to which sedative drugs interfere with MER is unknown. We recorded the population activity of STN neurons during propofol sedation and examined its effect on neuronal activity.

METHODS

The procedure was performed during DBS surgery for Parkinson disease. We administered propofol (50 μg/kg/min) at a constant electrode location in the STN until stable sedation was achieved. We recorded the electrical activity, and calculated its root mean square (RMS) before, during, and after the propofol infusions.

RESULTS

The activity of 24 electrode trajectories was recorded in 16 patients. The RMS of STN activity decreased significantly after propofol administration in 18 of the 24 trajectories. The average normalized RMS decreased by 23.2%± 9.1% (mean ± SD) during propofol administration (P < 0.001), and returned to baseline 9.3 ± 4.0 minutes after it was stopped.

CONCLUSIONS

Propofol administration leads to a significant decrease of STN neuronal activity. Thus, it may interfere with MER identification of the STN borders. However, activity returns to baseline shortly after administration stops. Therefore, propofol can be safely used until shortly before MER for DBS.