The impact of visual gaze direction on auditory object tracking

10 Hz rhythmic stimulation modulates electrophysiological, but not behavioral markers of suppression

We investigated the role of alpha in the suppression of attention capture by salient but to-be-suppressed (negative and nonpredictive) color cues, expecting a potential boosting effect of alpha-rhythmic entrainment on feature-specific cue suppression. We did so by presenting a rhythmically flickering visual bar of 10 Hz before the cue - either on the cue's side or opposite the cue -while an arrhythmically flickering visual bar was presented on the respective other side. We hypothesized that rhythmic entrainment at cue location could enhance the suppression of the cue. Testing 27 participants ranging from 18 to 39 years of age, we found both behavioral and electrophysiological evidence of suppression: Search times for a target at a negatively cued location were delayed relative to a target away from the cued location (inverse validity effects). In addition, an event-related potential indicative for suppression (the Distractor Positivity, Pd) was observed following rhythmic but not arrhythmic stimulation, indicating that suppression was boosted by the stimulation. This was also echoed in higher spectral power and intertrial phase coherence of EEG at rhythmically versus arrhythmically stimulated electrode sites, albeit only at the second harmonic (20 Hz), but not at the stimulation frequency. In addition, inverse validity effects were not modulated by rhythmic entrainment congruent with the cue side. Hence, we propose that rhythmic visual stimulation in the alpha range could support suppression, though behavioral evidence remains elusive, in contrast to electrophysiological findings.

Eye movements track prioritized auditory features in selective attention to natural speech

Over the last decades, cognitive neuroscience has identified a distributed set of brain regions that are critical for attention. Strong anatomical overlap with brain regions critical for oculomotor processes suggests a joint network for attention and eye movements. However, the role of this shared network in complex, naturalistic environments remains understudied. Here, we investigated eye movements in relation to (un)attended sentences of natural speech. Combining simultaneously recorded eye tracking and magnetoencephalographic data with temporal response functions, we show that gaze tracks attended speech, a phenomenon we termed ocular speech tracking. Ocular speech tracking even differentiates a target from a distractor in a multi-speaker context and is further related to intelligibility. Moreover, we provide evidence for its contribution to neural differences in speech processing, emphasizing the necessity to consider oculomotor activity in future research and in the interpretation of neural differences in auditory cognition.

What are wereallydecoding? Unveiling biases in EEG-based decoding of the spatial focus of auditory attention

Objective.Spatial auditory attention decoding (Sp-AAD) refers to the task of identifying the direction of the speaker to which a person is attending in a multi-talker setting, based on the listener's neural recordings, e.g. electroencephalography (EEG). The goal of this study is to thoroughly investigate potential biases when training such Sp-AAD decoders on EEG data, particularly eye-gaze biases and latent trial-dependent confounds, which may result in Sp-AAD models that decode eye-gaze or trial-specific fingerprints rather than spatial auditory attention.Approach.We designed a two-speaker audiovisual Sp-AAD protocol in which the spatial auditory and visual attention were enforced to be either congruent or incongruent, and we recorded EEG data from sixteen participants undergoing several trials recorded at distinct timepoints. We trained a simple linear model for Sp-AAD based on common spatial patterns filters in combination with either linear discriminant analysis (LDA) or k-means clustering, and evaluated them both across- and within-trial.Main results.We found that even a simple linear Sp-AAD model is susceptible to overfitting to confounding signal patterns such as eye-gaze and trial fingerprints (e.g. due to feature shifts across trials), resulting in artificially high decoding accuracies. Furthermore, we found that changes in the EEG signal statistics across trials deteriorate the trial generalization of the classifier, even when the latter is retrained on the test trial with an unsupervised algorithm.Significance.Collectively, our findings confirm that there exist subtle biases and confounds that can strongly interfere with the decoding of spatial auditory attention from EEG. It is expected that more complicated non-linear models based on deep neural networks, which are often used for Sp-AAD, are even more vulnerable to such biases. Future work should perform experiments and model evaluations that avoid and/or control for such biases in Sp-AAD tasks.

The influence of hatha yoga on stress, anxiety, and suppression: A randomized controlled trial

Engaging in yoga may mitigate stress and anxiety in individuals while potentially enhancing one's capacity to manage distractions. Our research aimed to explore the relation between these two outcomes: Can an eight-week yoga program foster distraction suppression, thereby reducing stress and discomfort? To answer this question, we used Hatha Yoga, the most commonly practiced form of yoga. We tested if the intervention improved participants' ability to suppress distractions and selectively decrease self-reported stress and stress reactivity. In Addition, we investigated whether such an intervention would increase participants' mindfulness. Our study included 98 healthy yoga novices between 18 and 40 years who were randomly assigned to either an experimental or a waitlist condition, with each participant completing pre- and post-intervention assessments, including questionnaires, as well as electrophysiological and behavioral measures. After eight weeks of yoga practice, significant reductions in self-reported stress and stress reactivity levels, as well as increased mindfulness, were observed among those participating in the intervention relative to those in the waitlist control group. There were, however, no significant changes in state or trait anxiety due to the intervention. Changes in stress measures could not be explained by changes in participants' ability to suppress distractors, which was not affected by the intervention. Overall, our findings suggest that regular participation in Hatha Yoga can improve mental health outcomes without impacting cognitive functioning directly related to distractor suppression. CLINICAL TRIAL REGISTRATION NUMBER: NCT05232422.

An Effect of Gaze Direction in Cocktail Party Listening

It is well established that gaze direction can influence auditory spatial perception, but the implications of this interaction for performance in complex listening tasks is unclear. In the current study, we investigated whether there is a measurable effect of gaze direction on speech intelligibility in a "cocktail party" listening situation. We presented sequences of digits from five loudspeakers positioned at 0°,  ±  15°, and ± 30° azimuth, and asked participants to repeat back the digits presented from a designated target loudspeaker. In different blocks of trials, the participant visually fixated on a cue presented at the target location or at a nontarget location. Eye position was tracked continuously to monitor compliance. Performance was best when fixation was on-target (vs. off-target) and the size of this effect depended on the specific configuration. This result demonstrates an influence of gaze direction in multitalker mixtures, even in the absence of visual speech information.

Comparing In-ear EOG for Eye-Movement Estimation With Eye-Tracking: Accuracy, Calibration, and Speech Comprehension

This presentation details and evaluates a method for estimating the attended speaker during a two-person conversation by means of in-ear electro-oculography (EOG). Twenty-five hearing-impaired participants were fitted with molds equipped with EOG electrodes (in-ear EOG) and wore eye-tracking glasses while watching a video of two life-size people in a dialog solving a Diapix task. The dialogue was directionally presented and together with background noise in the frontal hemisphere at 60 dB SPL. During three conditions of steering (none, in-ear EOG, conventional eye-tracking), participants' comprehension was periodically measured using multiple-choice questions. Based on eye movement detection by in-ear EOG or conventional eye-tracking, the estimated attended speaker was amplified by 6 dB. In the in-ear EOG condition, the estimate was based on one selected channel pair of electrodes out of 36 possible electrodes. A novel calibration procedure introducing three different metrics was used to select the measurement channel. The in-ear EOG attended speaker estimates were compared to those of the eye-tracker. Across participants, the mean accuracy of in-ear EOG estimation of the attended speaker was 68%, ranging from 50 to 89%. Based on offline simulation, it was established that higher scoring metrics obtained for a channel with the calibration procedure were significantly associated with better data quality. Results showed a statistically significant improvement in comprehension of about 10% in both steering conditions relative to the no-steering condition. Comprehension in the two steering conditions was not significantly different. Further, better comprehension obtained under the in-ear EOG condition was significantly correlated with more accurate estimation of the attended speaker. In conclusion, this study shows promising results in the use of in-ear EOG for visual attention estimation with potential for applicability in hearing assistive devices.

Visual predictions, neural oscillations and naïve physics

Prediction is a core function of the human visual system. Contemporary research suggests the brain builds predictive internal models of the world to facilitate interactions with our dynamic environment. Here, we wanted to examine the behavioural and neurological consequences of disrupting a core property of peoples’ internal models, using naturalistic stimuli. We had people view videos of basketball and asked them to track the moving ball and predict jump shot outcomes, all while we recorded eye movements and brain activity. To disrupt people’s predictive internal models, we inverted footage on half the trials, so dynamics were inconsistent with how movements should be shaped by gravity. When viewing upright videos people were better at predicting shot outcomes, at tracking the ball position, and they had enhanced alpha-band oscillatory activity in occipital brain regions. The advantage for predicting upright shot outcomes scaled with improvements in ball tracking and occipital alpha-band activity. Occipital alpha-band activity has been linked to selective attention and spatially-mapped inhibitions of visual brain activity. We propose that when people have a more accurate predictive model of the environment, they can more easily parse what is relevant, allowing them to better target irrelevant positions for suppression—resulting in both better predictive performance and in neural markers of inhibited information processing.

Cross-modality modulation of auditory midbrain processing of intensity information

In nature, animals constantly receive a multitude of sensory stimuli, such as visual, auditory, and somatosensory. The integration across sensory modalities is advantageous for the precise processing of sensory inputs which is essential for animals to survival. Although some principles of cross-modality integration have been revealed by many studies, little insight has been gained into its functional potentials. In this study, the functional influence of cross-modality modulation on auditory processing of intensity information was investigated via recording neuronal activity in the auditory midbrain (i.e., inferior colliculus, IC) under the conditions of visual, auditory, and audiovisual stimuli, respectively. Results demonstrated that combined audiovisual stimuli either enhanced or suppressed the responses of IC neurons compared to auditory stimuli alone, even though the same visual stimuli alone induced no response. Audiovisual modulation appeared to be strongest when the combined audiovisual stimuli were located at the best auditory azimuth of neurons as well as when presented with intensity at near-threshold levels. Additionally, the rate-intensity function of IC neurons to auditory stimuli was expanded or compressed by audiovisual modulation, which was highly dependent on the minimal threshold (MT) of neurons. Lowering of the MT and greater audiovisual modulation for the neuron indicated an intensity-specific enhancement of auditory intensity sensitivity by cross-modality modulation. Overall, evidence suggests a potential functional role of cross-modality modulation in IC that serves to instruct adaptive plasticity to enhance the auditory perception of intensity information.

Temporal contrast effects in human speech perception are immune to selective attention

Two fundamental properties of perception are selective attention and perceptual contrast, but how these two processes interact remains unknown. Does an attended stimulus history exert a larger contrastive influence on the perception of a following target than unattended stimuli? Dutch listeners categorized target sounds with a reduced prefix "ge-" marking tense (e.g., ambiguous between gegaan-gaan "gone-go"). In 'single talker' Experiments 1-2, participants perceived the reduced syllable (reporting gegaan) when the target was heard after a fast sentence, but not after a slow sentence (reporting gaan). In 'selective attention' Experiments 3-5, participants listened to two simultaneous sentences from two different talkers, followed by the same target sounds, with instructions to attend only one of the two talkers. Critically, the speech rates of attended and unattended talkers were found to equally influence target perception - even when participants could watch the attended talker speak. In fact, participants' target perception in 'selective attention' Experiments 3-5 did not differ from participants who were explicitly instructed to divide their attention equally across the two talkers (Experiment 6). This suggests that contrast effects of speech rate are immune to selective attention, largely operating prior to attentional stream segregation in the auditory processing hierarchy.

Effect of feedback and target size on eye gaze accuracy in an off-screen task

PURPOSE

Eye gaze interfaces have been used by people with severe physical impairment to interact with various assistive technologies. If used to control robots, it would be beneficial if individuals could gaze directly at targets in the physical environment rather than have to switch their gaze between a screen with representations of robot commands and the physical environment to see the response of their selection. By using a homogeneous transformation technique, eye gaze coordinates can be mapped between the reference coordinate frame of eye tracker and the coordinate frame of objects in the physical environment. Feedback about where the eye tracker has determined the eye gaze is fixated is needed so users can select targets more accurately. Screen-based assistive technologies can use visual feedback, but in a physical environment, other forms of feedback need to be examined.

MATERIALS AND METHODS

In this study, an eye gaze system with different feedback conditions (i.e., visual, auditory, vibrotactile, and no-feedback) was tested when participants received visual feedback on a display (on-screen) and when looking directly at the physical environment (off-screen). Target selection tasks in both screen conditions were performed by ten non-disabled adults, three non-disabled children, and two adults and one child with cerebral palsy.

RESULTS

Tasks performed with gaze fixation feedback modalities were accomplished faster and with higher success than tasks performed without feedback, and similar results were observed in both screen conditions. No significant difference was observed in performance across the feedback modalities, but participants had personal preferences.

CONCLUSION

The homogeneous transformation technique enabled the use of a stationary eye tracker to select target objects in the physical environment, and auditory and vibrotactile feedback enabled participants to be more accurate selecting targets than without it.Implications for RehabilitationBeing able to select target objects in the physical environment by eye gaze could make it easier for children with disabilities to control assistive robots, because in this way they do not have to change their focus between a computer screen with commands and the robot.Providing auditory or vibrotactile feedback when using an eye gaze system made it faster and easier to know if a target was being gazed upon.Being able to select targets in the environment using eye gaze could be beneficial for other assistive technology, too, such as destination selection for power wheelchairs.

Examining the Role of Eye Movements During Conversational Listening in Noise

Speech comprehension is often thought of as an entirely auditory process, but both normal hearing and hearing-impaired individuals sometimes use visual attention to disambiguate speech, particularly when it is difficult to hear. Many studies have investigated how visual attention (or the lack thereof) impacts the perception of simple speech sounds such as isolated consonants, but there is a gap in the literature concerning visual attention during natural speech comprehension. This issue needs to be addressed, as individuals process sounds and words in everyday speech differently than when they are separated into individual elements with no competing sound sources or noise. Moreover, further research is needed to explore patterns of eye movements during speech comprehension – especially in the presence of noise – as such an investigation would allow us to better understand how people strategically use visual information while processing speech. To this end, we conducted an experiment to track eye-gaze behavior during a series of listening tasks as a function of the number of speakers, background noise intensity, and the presence or absence of simulated hearing impairment. Our specific aims were to discover how individuals might adapt their oculomotor behavior to compensate for the difficulty of the listening scenario, such as when listening in noisy environments or experiencing simulated hearing loss. Speech comprehension difficulty was manipulated by simulating hearing loss and varying background noise intensity. Results showed that eye movements were affected by the number of speakers, simulated hearing impairment, and the presence of noise. Further, findings showed that differing levels of signal-to-noise ratio (SNR) led to changes in eye-gaze behavior. Most notably, we found that the addition of visual information (i.e. videos vs. auditory information only) led to enhanced speech comprehension – highlighting the strategic usage of visual information during this process.

Absolute Eye Gaze Estimation With Biosensors in Hearing Aids

People with hearing impairment typically have difficulties following conversations in multi-talker situations. Previous studies have shown that utilizing eye gaze to steer audio through beamformers could be a solution for those situations. Recent studies have shown that in-ear electrodes that capture electrooculography in the ear (EarEOG) can estimate the eye-gaze relative to the head, when the head was fixed. The head movement can be estimated using motion sensors around the ear to create an estimate of the absolute eye-gaze in the room. In this study, an experiment was designed to mimic a multi-talker situation in order to study and model the EarEOG signal when participants attempted to follow a conversation. Eleven hearing impaired participants were presented speech from the DAT speech corpus (Bo Nielsen et al., 2014), with three targets positioned at -30°, 0° and +30° azimuth. The experiment was run in two setups: one where the participants had their head fixed in a chinrest, and the other where they were free to move their head. The participants' task was to focus their visual attention on an LED-indicated target that changed regularly. A model was developed for the relative eye-gaze estimation, taking saccades, fixations, head movement and drift from the electrode-skin half-cell into account. This model explained 90.5% of the variance of the EarEOG when the head was fixed, and 82.6% when the head was free. The absolute eye-gaze was also estimated utilizing that model. When the head was fixed, the estimation of the absolute eye-gaze was reliable. However, due to hardware issues, the estimation of the absolute eye-gaze when the head was free had a variance that was too large to reliably estimate the attended target. Overall, this study demonstrated the potential of estimating absolute eye-gaze using EarEOG and motion sensors around the ear.

Visual input shapes the auditory frequency responses in the inferior colliculus of mouse

The integration of visual and auditory information is important for humans or animals to build an accurate and coherent perception of the external world. Although some evidence has shown some principles of the audiovisual integration, little insight has been gained into its functional purpose. In this study, we investigated the functional influence of dynamic visual input on auditory frequency processing by recording single unit activity in the central nucleus of the inferior colliculus (ICc). Results showed that the auditory responses of ICc neurons to sound frequencies could be enhanced or suppressed by visual stimuli even though the same visual stimuli induced no neural responses when presented alone. For each ICc neuron, the most effective visual stimuli were located in the same azimuth as for auditory stimuli and preceded for ∼20 ms. Additionally, visual stimuli could steepen or flatten the frequency tuning curves (FTCs) of ICc neurons by various visual effects at each responsive frequency. The modulation degree of auditory FTCs was dependent on the minimal thresholds (MTs) of ICc neurons, i.e., with MTs increasing, the modulation degree decreased. Due to the non-homogeneous distribution of MTs which was lowest at 10 kHz, visual modulation of auditory FTCs exhibited a frequency-specific manner, the closer it reached the characteristic frequency (CF) of 10 kHz, the greater modulation. Thus, visual modulation of auditory frequency responses in ICc is dependent not only on the visual stimulus but also on the auditory characteristics of ICc neurons. These results suggest a moment-to-moment visual modulation of auditory frequency responses that in real time increase auditory frequency sensitivity to audiovisual stimuli. Furthermore, in the long term such modulation could serve to instruct auditory adaptive plasticity to maintain necessary and accurate auditory detection and perceptual behavior.

Contralateral delay activity during temporal order memory

In everyday life, we constantly need to remember the temporal sequence of visual events over short periods of time, for example, when making sense of others' actions or watching a movie. While there is increasing knowledge available on neural mechanisms underlying visual working memory (VWM) regarding the identity and spatial location of objects, less is known about how the brain encodes and retains information on temporal sequences. Here, we investigate whether the contralateral-delay activity (CDA), a well-studied electroencephalographic (EEG) component associated with VWM of object identity, also reflects the encoding and retention of temporal order. In two independent experiments, we presented participants with a sequence of four or six images, followed by a 1 s retention period. Participants judged temporal order by indicating whether a subsequently presented probe image was originally displayed during the first or the second half of the sequence. As a main novel result, we report the emergence of a contralateral negativity already following the presentation of the first item of the sequence, which increases over the course of a trial with every presented item, up to a limit of four items. We further observed no differences in the CDA during the temporal-order task compared to one obtained during a task concerning the spatial location of the presented items. Since the characteristics of the CDA appear to be highly similar between different encoded feature dimensions and increases as additional items are being encoded, we suggest this component might be a general characteristic of various types of VWM.

Effect of Visuospatial Attention on the Sensorimotor Gating System

The integration of multiple sensory modalities allows us to adapt to the environment of the outside world. It is widely known that visual stimuli interfere with the processing of auditory information, which is involved in the ability to pay attention. Additionally, visuospatial attention has the characteristic of laterality. It is unclear whether this laterality of visuospatial attention affects the processing of auditory stimuli. The sensorimotor gating system is a neurological process, which filters out unnecessary stimuli from environmental stimuli in the brain. Prepulse inhibition (PPI) is an operational measure of the sensorimotor gating system, which a weaker prestimulus (prepulse), such as a visual stimulus, inhibits the startle reflex elicited by a subsequent robust startling stimulus (pulse) such as a tone. Therefore, we investigated whether the visual stimulus from the left or right visual space affects the sensorimotor gating system in a "rest" task (low attentional condition) and a "selective attention" task (high attentional condition). In the selective attention task, we found that the target prepulse presented in the left and bilateral visual fields suppressed the startle reflex more than that presented in the right visual field. By contrast, there was no laterality of PPI in the no-target prepulse condition, and there was no laterality of PPI in the rest task. These results suggest that the laterality of visuospatial attention affects the sensorimotor gating system depending on the attentional condition. Moreover, the process of visual information processing may differ between the left and right brain.

Late cortical tracking of ignored speech facilitates neural selectivity in acoustically challenging conditions

Listening requires selective neural processing of the incoming sound mixture, which in humans is borne out by a surprisingly clean representation of attended-only speech in auditory cortex. How this neural selectivity is achieved even at negative signal-to-noise ratios (SNR) remains unclear. We show that, under such conditions, a late cortical representation (i.e., neural tracking) of the ignored acoustic signal is key to successful separation of attended and distracting talkers (i.e., neural selectivity). We recorded and modeled the electroencephalographic response of 18 participants who attended to one of two simultaneously presented stories, while the SNR between the two talkers varied dynamically between +6 and -6 dB. The neural tracking showed an increasing early-to-late attention-biased selectivity. Importantly, acoustically dominant (i.e., louder) ignored talkers were tracked neurally by late involvement of fronto-parietal regions, which contributed to enhanced neural selectivity. This neural selectivity, by way of representing the ignored talker, poses a mechanistic neural account of attention under real-life acoustic conditions.