Multisensory signalling enhances pupil dilation

Smooth pursuit inhibition reveals audiovisual enhancement of fast movement control

The sudden onset of a visual object or event elicits an inhibition of eye movements at latencies approaching the minimum delay of visuomotor conductance in the brain. Typically, information presented via multiple sensory modalities, such as sound and vision, evokes stronger and more robust responses than unisensory information. Whether and how multisensory information affects ultra-short latency oculomotor inhibition is unknown. In two experiments, we investigate smooth pursuit and saccadic inhibition in response to multisensory distractors. Observers tracked a horizontally moving dot and were interrupted by an unpredictable visual, auditory, or audiovisual distractor. Distractors elicited a transient inhibition of pursuit eye velocity and catch-up saccade rate within ∼100 ms of their onset. Audiovisual distractors evoked stronger oculomotor inhibition than visual- or auditory-only distractors, indicating multisensory response enhancement. Multisensory response enhancement magnitudes were equal to the linear sum of responses to component stimuli. These results demonstrate that multisensory information affects eye movements even at ultra-short latencies, establishing a lower time boundary for multisensory-guided behavior. We conclude that oculomotor circuits must have privileged access to sensory information from multiple modalities, presumably via a fast, subcortical pathway.

How the eyes respond to sounds

Eye movements have been extensively studied with respect to visual stimulation. However, we live in a multisensory world, and how the eyes are driven by other senses has been explored much less. Here, we review the evidence on how audition can trigger and drive different eye responses and which cortical and subcortical neural correlates are involved. We provide an overview on how different types of sounds, from simple tones and noise bursts to spatially localized sounds and complex linguistic stimuli, influence saccades, microsaccades, smooth pursuit, pupil dilation, and eye blinks. The reviewed evidence reveals how the auditory system interacts with the oculomotor system, both behaviorally and neurally, and how this differs from visually driven eye responses. Some evidence points to multisensory interaction, and potential multisensory integration, but the underlying computational and neural mechanisms are still unclear. While there are marked differences in how the eyes respond to auditory compared to visual stimuli, many aspects of auditory-evoked eye responses remain underexplored, and we summarize the key open questions for future research.

Linear integration of multisensory signals in the pupil

The pupil of the eye responds to various salient signals from different modalities, but there is no consensus on how these pupillary responses are integrated when multiple signals appear simultaneously. Both linear and nonlinear integration have been found previously. The current study aimed to reexamine the nature of pupillary integration, and specifically focused on the early, transient pupillary responses due to its close relationship with orienting. To separate the early pupillary responses out of the pupil time series, we adopted a pupil oscillation paradigm in which sensory stimuli were periodically presented. The simulation analysis confirmed that the amplitude of the pupil oscillation, induced by stimuli repeatedly presented at relatively high rates, can precisely reflect the early, transient pupillary responses without involving the late and sustained pupillary responses. The experimental results then showed that the amplitude of pupil oscillation induced by a series of simultaneous audiovisual stimuli equaled to a linear summation of the oscillatory amplitudes when unisensory stimuli were presented alone. Moreover, the tonic arousal levels, indicated by the baseline pupil size, cannot shift the summation from linear to nonlinear. These findings together support the additive nature of multisensory pupillary integration for the early, orienting-related pupillary responses. The additive nature of pupillary integration further implies that multiple pupillary responses may be independent of each other, irrespective of their potential cognitive and neural drivers.

Evidence that pupil dilation and cardiac afferent signalling differentially impact the processing of emotional intensity and racial bias

Interoceptive cardiac arousal signals (e.g., from baroreceptor firing at ventricular systole compared to diastole) have been found to enhance perception of fearful versus neutral faces. They have also been found to amplify racially biased misidentification of tools as weapons when preceded by facial images of Black versus White individuals. Since pupil size is strongly coupled to arousal, we tested if experimental manipulation of pupil size influences fear processing in emotional judgement and racial bias tasks involving measurement of cardiac signals. In a sample of 22 non-clinical participants in an emotional intensity judgement task, pupil size did not affect emotional intensity ratings. Nor did it interact with differential effects of cardiac systole versus diastole on intensity judgements of fearful and neutral faces, replicated here. In a sample of 25 non-clinical participants in a weapons identification task, larger pupil size resulted in faster response times and lower accuracy when identifying tools and weapons. However, pupil size did not interact with weapon versus tool identification, race of prime, or cardiac timing. We nevertheless replicated the observed increase in racially biased misidentification of tools as weapons following Black face primes presented at cardiac systole. Together our findings indicate that pupil dilation does not directly influence the processing of fear cues or perceived threat (as in racial bias) yet affects task performance by decreasing response times and accuracy. These findings contrast with the established effect of cardiac arousal signals on threat processing and may help focus interventions to mitigate related decision errors in high-pressure occupations.

Multisensory decisions from self to world

Classic Bayesian models of perceptual inference describe how an ideal observer would integrate 'unisensory' measurements (multisensory integration) and attribute sensory signals to their origin(s) (causal inference). However, in the brain, sensory signals are always received in the context of a multisensory bodily state-namely, in combination with other senses. Moreover, sensory signals from both interoceptive sensing of one's own body and exteroceptive sensing of the world are highly interdependent and never occur in isolation. Thus, the observer must fundamentally determine whether each sensory observation is from an external (versus internal, self-generated) source to even be considered for integration. Critically, solving this primary causal inference problem requires knowledge of multisensory and sensorimotor dependencies. Thus, multisensory processing is needed to separate sensory signals. These multisensory processes enable us to simultaneously form a sense of self and form distinct perceptual decisions about the external world. In this opinion paper, we review and discuss the similarities and distinctions between multisensory decisions underlying the sense of self and those directed at acquiring information about the world. We call attention to the fact that heterogeneous multisensory processes take place all along the neural hierarchy (even in forming 'unisensory' observations) and argue that more integration of these aspects, in theory and experiment, is required to obtain a more comprehensive understanding of multisensory brain function. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

A virtual reality study investigating the train illusion

The feeling of self-movement that occurs in the absence of physical motion is often referred to as vection, which is commonly exemplified using the train illusion analogy (TIA). Limited research exists on whether the TIA accurately exemplifies the experience of vection in virtual environments (VEs). Few studies complemented their vection research with participants' qualitative feedback or by recording physiological responses, and most studies used stimuli that contextually differed from the TIA. We investigated whether vection is experienced differently in a VE replicating the TIA compared to a VE depicting optic flow by recording subjective and physiological responses. Additionally, we explored participants' experience through an open question survey. We expected the TIA environment to induce enhanced vection compared to the optic flow environment. Twenty-nine participants were visually and audibly immersed in VEs that either depicted optic flow or replicated the TIA. Results showed optic flow elicited more compelling vection than the TIA environment and no consistent physiological correlates to vection were identified. The post-experiment survey revealed discrepancies between participants' quantitative and qualitative feedback. Although the dynamic content may outweigh the ecological relevance of the stimuli, it was concluded that more qualitative research is needed to understand participants' vection experience in VEs.

Pupillary responses to differences in luminance, color and set size

The pupil responds to a salient stimulus appearing in the environment, in addition to its modulation by global luminance. These pupillary responses can be evoked by visual or auditory stimuli, scaled with stimulus salience, and enhanced by multisensory presentation. In addition, pupil size is modulated by various visual stimulus attributes, such as color, area, and motion. However, research that concurrently examines the influence of different factors on pupillary responses is limited. To explore how presentation of multiple visual stimuli influences human pupillary responses, we presented arrays of visual stimuli and systematically varied their luminance, color, and set size. Saliency level, computed by the saliency model, systematically changed with set size across all conditions, with higher saliency levels in larger set sizes. Pupillary constriction responses were evoked by the appearance of visual stimuli, with larger pupillary responses observed in larger set size. These effects were pronounced even though the global luminance level was unchanged using isoluminant chromatic stimuli. Furthermore, larger pupillary constriction responses were obtained in the blue, compared to other color conditions. Together, we argue that both cortical and subcortical areas contribute to the observed pupillary constriction modulated by set size and color.

Multisensory signals inhibit pupillary light reflex: Evidence from pupil oscillation

Multisensory integration, which enhances stimulus saliency at the early stage of the processing hierarchy, has been recently shown to produce a larger pupil size than its unisensory constituents. Theoretically, any modulation on pupil size ought to be associated with the sympathetic and parasympathetic pathways that are sensitive to light. But it remains poorly understood how the pupillary light reflex is changed in a multisensory context. The present study evoked an oscillation of the pupillary light reflex by periodically changing the luminance of a visual stimulus at 1.25 Hz. It was found that such induced pupil size oscillation was substantially attenuated when the bright but not the dark phase of the visual flicker was periodically and synchronously presented with a burst of tones. This inhibition effect persisted when the visual flicker was task-irrelevant and out of attentional focus, but disappeared when the visual flicker was moved from the central field to the periphery. These findings not only offer a comprehensive characterization of the multisensory impact on pupil response to light, but also provide valuable clues about the individual contributions of the sympathetic and parasympathetic pathways to multisensory modulation of pupil size.

Role of the frontal eye field in human pupil and saccade orienting responses

The appearance of a salient stimulus evokes a series of orienting responses including saccades and pupil size to prepare the body for appropriate action. The midbrain superior colliculus (SC) that receives critical control signals from the frontal eye field (FEF) is hypothesized to coordinate all components of orienting. It has shown recently that the FEF, together with the SC, is also importantly involved in the control of pupil size, in addition to its well-documented role in eye movements. Although the role of the FEF in pupil size is demonstrated in monkeys, its role in human pupil responses and the coordination between pupil size and saccades remains to be established. Through applying continuous theta-burst stimulation over the right FEF and vertex, we investigated the role of the FEF in human pupil and saccade responses evoked by a salient stimulus, and the coordination between pupil size and saccades. Our results showed that neither saccade reaction times (SRT) nor pupil responses evoked by salient stimuli were modulated by FEF stimulation. In contrast, the correlation between pupil size and SRTs in the contralateral stimulus condition was diminished with FEF stimulation, but intact with vertex stimulation. Moreover, FEF stimulation effects between saccade and pupil responses associated with salient stimuli correlated across participants. This is the first transcranial magnetic stimulation (TMS) study on the pupil orienting response, and our findings suggest that human FEF was involved in coordinating pupil size and saccades, but not involved in the control of pupil orienting responses.

The additive nature of the human multisensory evoked pupil response

Pupillometry has received increased interest for its usefulness in measuring various sensory processes as an alternative to behavioural assessments. This is also apparent for multisensory investigations. Studies of the multisensory pupil response, however, have produced conflicting results. Some studies observed super-additive multisensory pupil responses, indicative of multisensory integration (MSI). Others observed additive multisensory pupil responses even though reaction time (RT) measures were indicative of MSI. Therefore, in the present study, we investigated the nature of the multisensory pupil response by combining methodological approaches of previous studies while using supra-threshold stimuli only. In two experiments we presented auditory and visual stimuli to observers that evoked a(n) (onset) response (be it constriction or dilation) in a simple detection task and a change detection task. In both experiments, the RT data indicated MSI as shown by race model inequality violation. Still, the multisensory pupil response in both experiments could best be explained by linear summation of the unisensory pupil responses. We conclude that the multisensory pupil response for supra-threshold stimuli is additive in nature and cannot be used as a measure of MSI, as only a departure from additivity can unequivocally demonstrate an interaction between the senses.

EMG-Free Monitorization of the Acoustic Startle Reflex with a Mobile Phone: Implications of Sound Parameters with Posture Related Responses

(1) Background: Acute acoustic (sound) stimulus prompts a state of defensive motivation in which unconscious muscle responses are markedly enhanced in humans. The orbicularis oculi (OO) of the eye is an easily accessed muscle common for acoustic startle reaction/response/reflex (ASR) investigations and is the muscle of interest in this study. Although the ASR can provide insights about numerous clinical conditions, existing methodologies (Electromyogram, EMG) limit the usability of the method in real clinical conditions. (2) Objective: With EMG-free muscle recording in mind, our primary aim was to identify and investigate potential correlations in the responses of individual and cooperative OO muscles to various acoustic stimuli using a mobile and wire-free system. Our secondary aim was to investigate potential altered responses to high and also relatively low intensity acoustics at different frequencies in both sitting and standing positions through the use of biaural sound induction and video diagnostic techniques and software. (3) Methods: This study used a mobile-phone acoustic startle response monitoring system application to collect blink amplitude and velocity data on healthy males, aged 18–28 community cohorts during (n = 30) in both sitting and standing postures. The iPhone X application delivers specific sound parameters and detects blinking responses to acoustic stimulus (in millisecond resolution) to study the responses of the blinking reflex to acoustic sounds in standing and sitting positions by using multiple acoustic test sets of different frequencies and amplitudes introduced as acute sound stimuli (<0.5 s). The single acoustic battery of 15 pure-square wave sounds consisted of frequencies and amplitudes between 500, 1000, 2000, 3000, and 4000 Hz scales using 65, 90, and 105 dB (e.g., 3000 Hz_90 dB). (4) Results: Results show that there was a synchronization of amplitude and velocity between both eyes to all acoustic startles. Significant differences (p = 0.01) in blinking reaction time between sitting vs. standing at the high intensity (105 dB) 500 Hz acoustic test set was discovered. Interestingly, a highly significant difference (p < 0.001) in response times between test sets 500 Hz_105 dB and 4000 Hz_105 dB was identified. (5) Conclusions: To our knowledge, this is the first mobile phone-based acoustic battery used to detect and report significant ASR responses to specific frequencies and amplitudes of sound stimulus with corresponding sitting and standing conditions. The results from this experiment indicate the potential significance of using the specific frequency, amplitude, and postural conditions (as never before identified) which can open new horizons for ASR to be used for diagnosis and monitoring in numerous clinical and remote or isolated conditions.

Saliency and priority modulation in a pop-out paradigm: Pupil size and microsaccades

A salient stimulus can trigger a coordinated orienting response consisting of a saccade, pupil, and microsaccadic responses. Saliency models predict that the degree of visual conspicuity of all visual stimuli guides visual orienting. By presenting a multiple-item array that included an oddball colored item (pop-out), randomly mixed colored items (mixed-color), or single-color items (single-color), we examined the effects of saliency and priority (saliency + relevancy) on pupil size and microsaccade responses. Larger pupil responses were produced in the pop-out compared to the mixed-color or single-color conditions after stimulus presentation. However, the saliency modulation on microsaccades was not significant. Furthermore, although goal-relevancy information did not modulate pupil responses and microsaccade rate, microsaccade direction was biased toward the pop-out item when it was the subsequent saccadic target. Together, our results demonstrate saliency modulation on pupil size and priority effects on microsaccade direction during visual pop-out.

Pupil dilation to emotional expressions in adolescent social anxiety disorder is related to treatment outcome

Atypical attention to potential social threats, such as emotional faces, may be one of the core mechanisms underlying social anxiety disorder (SAD). Pupil dilation is an index of locus coreuleus-noradrenergic activity, and closely linked to attention. In the present study, pupil dilation was studied in adolescents with SAD (N = 26; 22 Female) before the onset of a 12-week cognitive behavioral treatment, and in healthy controls (N = 23). Stimuli were faces with angry or happy emotional expressions. Contrary to our hypothesis, the SAD group did not show hyper-responsiveness to angry compared to happy faces. Instead, an atypical time course of the pupil dilation response was found, resulting in an attenuated response during late time stages. Larger pupil dilation amplitude to happy faces before treatment was related to worse treatment response. These results contribute significantly to our understanding of the mechanisms underlying adolescent SAD.

Cortical modulation of pupillary function: systematic review

BACKGROUND

The pupillary light reflex is the main mechanism that regulates the pupillary diameter; it is controlled by the autonomic system and mediated by subcortical pathways. In addition, cognitive and emotional processes influence pupillary function due to input from cortical innervation, but the exact circuits remain poorly understood. We performed a systematic review to evaluate the mechanisms behind pupillary changes associated with cognitive efforts and processing of emotions and to investigate the cerebral areas involved in cortical modulation of the pupillary light reflex.

METHODOLOGY

We searched multiple databases until November 2018 for studies on cortical modulation of pupillary function in humans and non-human primates. Of 8,809 papers screened, 258 studies were included.

RESULTS

Most investigators focused on pupillary dilatation and/or constriction as an index of cognitive and emotional processing, evaluating how changes in pupillary diameter reflect levels of attention and arousal. Only few tried to correlate specific cerebral areas to pupillary changes, using either cortical activation models (employing micro-stimulation of cortical structures in non-human primates) or cortical lesion models (e.g., investigating patients with stroke and damage to salient cortical and/or subcortical areas). Results suggest the involvement of several cortical regions, including the insular cortex (Brodmann areas 13 and 16), the frontal eye field (Brodmann area 8) and the prefrontal cortex (Brodmann areas 11 and 25), and of subcortical structures such as the locus coeruleus and the superior colliculus.

CONCLUSIONS

Pupillary dilatation occurs with many kinds of mental or emotional processes, following sympathetic activation or parasympathetic inhibition. Conversely, pupillary constriction may occur with anticipation of a bright stimulus (even in its absence) and relies on a parasympathetic activation. All these reactions are controlled by subcortical and cortical structures that are directly or indirectly connected to the brainstem pupillary innervation system.

Opposing pupil responses to offered and anticipated reward values

Previous studies have shown that the pupils dilate more in anticipation of larger rewards. This finding raises the possibility of a more general association between reward amount and pupil size. We tested this idea by characterizing macaque pupil responses to offered rewards during evaluation and comparison in a binary choice task. To control attention, we made use of a design in which offers occurred in sequence. By looking at pupil responses after choice but before reward, we confirmed the previously observed positive association between pupil size and anticipated reward values. Surprisingly, however, we find that pupil size is negatively correlated with the value of offered gambles before choice, during both evaluation and comparison stages of the task. These results demonstrate a functional distinction between offered and anticipated rewards and present evidence against a narrow version of the simulation hypothesis; the idea that we represent offers by reactivating states associated with anticipating them. They also suggest that pupil size is correlated with relative, not absolute, values of offers, suggestive of an accept-reject model of comparison.

Heterosexual, Homosexual, and Bisexual Men's Pupillary Responses to Persons at Different Stages of Sexual Development

This study investigated whether pupil size during the viewing of images of adults and children reflects the sexual orientation of heterosexual, homosexual, and bisexual men (n = 100, M_age = 22). More specifically, we explored whether this measure corresponds with sexual age preferences for adults over children in nonpedophilic men. In general, results across three experiments, in which observers freely viewed or rated the sexual appeal of person images, suggest that pupil dilation to sexual stimuli is an indicator of sexual orientation toward adults. Heterosexual men's pupils dilated most strongly to adults of the other sex, homosexual men dilated most strongly to adults of the same sex, and bisexual men showed an intermediate pattern. Dilation to adults was substantially stronger than dilation to younger age groups. Sexual appeal ratings for images of adults and children also correlated with pupil responses, suggesting a direct link between pupil dilation and sexual interest. These findings provide support for pupil dilation as a measure of sex- and age-specific sexual preferences.

Neural, physiological, and behavioral correlates of visuomotor cognitive load

Visuomotor ability is quite crucial for everyday functioning, particularly in driving and sports. While there is accumulating evidence regarding neural correlates of visuomotor transformation, less is known about the brain regions that accommodate visuomotor mapping under different cognitive demands. We concurrently measured cortical activity and pupillary response, using functional near infrared spectroscopy (fNIRS) and eye-tracking glasses, to examine the neural systems linked to pupil dilation under varying cognitive demands. Twenty-three healthy adults performed two sessions of a navigation task, in which the cognitive load was manipulated by either reversing the visuomotor mapping or increasing the speed of the moving object. We identified a region in the right superior parietal lobule that responded to both types of visuomotor load and its activity was associated with larger pupillary response and better performance in the task. Our multimodal analyses suggest that activity in this region arises from the need for increased attentional effort and alertness for visuomotor control and is an ideal candidate for objective measurement of visuomotor cognitive load. Our data extend previous findings connecting changes in pupil diameter to neural activity under varying cognitive demand and have important implications for examining brain-behavior associations in real-world tasks such as driving and sports.