1
|
Gehmacher Q, Schubert J, Schmidt F, Hartmann T, Reisinger P, Rösch S, Schwarz K, Popov T, Chait M, Weisz N. Eye movements track prioritized auditory features in selective attention to natural speech. Nat Commun 2024; 15:3692. [PMID: 38693186 PMCID: PMC11063150 DOI: 10.1038/s41467-024-48126-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Accepted: 04/22/2024] [Indexed: 05/03/2024] Open
Abstract
Over the last decades, cognitive neuroscience has identified a distributed set of brain regions that are critical for attention. Strong anatomical overlap with brain regions critical for oculomotor processes suggests a joint network for attention and eye movements. However, the role of this shared network in complex, naturalistic environments remains understudied. Here, we investigated eye movements in relation to (un)attended sentences of natural speech. Combining simultaneously recorded eye tracking and magnetoencephalographic data with temporal response functions, we show that gaze tracks attended speech, a phenomenon we termed ocular speech tracking. Ocular speech tracking even differentiates a target from a distractor in a multi-speaker context and is further related to intelligibility. Moreover, we provide evidence for its contribution to neural differences in speech processing, emphasizing the necessity to consider oculomotor activity in future research and in the interpretation of neural differences in auditory cognition.
Collapse
Affiliation(s)
- Quirin Gehmacher
- Paris-Lodron-University of Salzburg, Department of Psychology, Centre for Cognitive Neuroscience, Salzburg, Austria.
| | - Juliane Schubert
- Paris-Lodron-University of Salzburg, Department of Psychology, Centre for Cognitive Neuroscience, Salzburg, Austria
| | - Fabian Schmidt
- Paris-Lodron-University of Salzburg, Department of Psychology, Centre for Cognitive Neuroscience, Salzburg, Austria
| | - Thomas Hartmann
- Paris-Lodron-University of Salzburg, Department of Psychology, Centre for Cognitive Neuroscience, Salzburg, Austria
| | - Patrick Reisinger
- Paris-Lodron-University of Salzburg, Department of Psychology, Centre for Cognitive Neuroscience, Salzburg, Austria
| | - Sebastian Rösch
- Department of Otorhinolaryngology, Head and Neck Surgery, Paracelsus Medical University Salzburg, 5020, Salzburg, Austria
| | | | - Tzvetan Popov
- Methods of Plasticity Research, Department of Psychology, University of Zurich, CH-8050, Zurich, Switzerland
- Department of Psychology, University of Konstanz, DE- 78464, Konstanz, Germany
| | - Maria Chait
- Ear Institute, University College London, London, UK
| | - Nathan Weisz
- Paris-Lodron-University of Salzburg, Department of Psychology, Centre for Cognitive Neuroscience, Salzburg, Austria
- Neuroscience Institute, Christian Doppler University Hospital, Paracelsus Medical University, Salzburg, Austria
| |
Collapse
|
2
|
Huang Y, Brosch M. Absence of eye position effects in the early auditory cortex of monkeys. Neuroreport 2024; 35:209-215. [PMID: 38251450 DOI: 10.1097/wnr.0000000000001985] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/23/2024]
Abstract
This study aims to investigate whether the position of the eyes affects the neuronal activity in auditory cortex in a condition in which not the active control of eye position but the execution of hand movements was required relative to stimuli. Two monkeys were trained to perform audio-visual tasks in which they had to use their hand to respond to both the visual and the auditory stimuli to earn a reward. We recorded the spiking activity and the local field potentials from the core fields of auditory cortex, along with the eye position of the monkeys while they performed the tasks. We found that both the spiking activity and the local field potentials did not significantly vary with the eye position. This was the case both during the presentation of sounds and during other periods of the tasks. Our results indicate that eye position did not affect the neuronal activity in auditory cortex during the audio-visual tasks. Our results, together with the previous finding that eye position affects the neuronal activity in auditory cortex during eye fixation tasks, suggest that the presence of eye position effects in auditory cortex depends on the specific behavior a subject has to exhibit to obtain a reward.
Collapse
Affiliation(s)
- Ying Huang
- Research Group Comparative Neuroscience, Leibniz Institute for Neurobiology
| | - Michael Brosch
- Research Group Comparative Neuroscience, Leibniz Institute for Neurobiology
- Center for Behavioral Brain Sciences, Otto-von-Guericke-University, Magdeburg, Germany
| |
Collapse
|
3
|
Hu J, Vetter P. How the eyes respond to sounds. Ann N Y Acad Sci 2024; 1532:18-36. [PMID: 38152040 DOI: 10.1111/nyas.15093] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2023]
Abstract
Eye movements have been extensively studied with respect to visual stimulation. However, we live in a multisensory world, and how the eyes are driven by other senses has been explored much less. Here, we review the evidence on how audition can trigger and drive different eye responses and which cortical and subcortical neural correlates are involved. We provide an overview on how different types of sounds, from simple tones and noise bursts to spatially localized sounds and complex linguistic stimuli, influence saccades, microsaccades, smooth pursuit, pupil dilation, and eye blinks. The reviewed evidence reveals how the auditory system interacts with the oculomotor system, both behaviorally and neurally, and how this differs from visually driven eye responses. Some evidence points to multisensory interaction, and potential multisensory integration, but the underlying computational and neural mechanisms are still unclear. While there are marked differences in how the eyes respond to auditory compared to visual stimuli, many aspects of auditory-evoked eye responses remain underexplored, and we summarize the key open questions for future research.
Collapse
Affiliation(s)
- Junchao Hu
- Visual and Cognitive Neuroscience Lab, Department of Psychology, University of Fribourg, Fribourg, Switzerland
| | - Petra Vetter
- Visual and Cognitive Neuroscience Lab, Department of Psychology, University of Fribourg, Fribourg, Switzerland
| |
Collapse
|
4
|
King CD, Lovich SN, Murphy DL, Landrum R, Kaylie D, Shera CA, Groh JM. Individual similarities and differences in eye-movement-related eardrum oscillations (EMREOs). Hear Res 2023; 440:108899. [PMID: 37979436 PMCID: PMC11081086 DOI: 10.1016/j.heares.2023.108899] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/03/2023] [Revised: 10/18/2023] [Accepted: 10/23/2023] [Indexed: 11/20/2023]
Abstract
We recently discovered a unique type of otoacoustic emission (OAE) time-locked to the onset (and offset) of saccadic eye movements and occurring in the absence of external sound (Gruters et al., 2018). How and why these eye-movement-related eardrum oscillations (EMREOs) are generated is unknown, with a role in visual-auditory integration being the likeliest candidate. Clues to both the drivers of EMREOs and their purpose can be gleaned by examining responses in normal hearing human subjects. Do EMREOs occur in all individuals with normal hearing? If so, what components of the response occur most consistently? Understanding which attributes of EMREOs are similar across participants and which show more variability will provide the groundwork for future comparisons with individuals with hearing abnormalities affecting the ear's various motor components. Here we report that in subjects with normal hearing thresholds and normal middle ear function, all ears show (a) measurable EMREOs (mean: 58.7 dB SPL; range 45-67 dB SPL for large contralateral saccades), (b) a phase reversal for contra- versus ipsilaterally-directed saccades, (c) a large peak in the signal occurring soon after saccade onset, (d) an additional large peak time-locked to saccade offset and (e) evidence that saccade duration is encoded in the signal. We interpret the attributes of EMREOs that are most consistent across subjects as the ones that are most likely to play an essential role in their function. The individual differences likely reflect normal variation in individuals' auditory system anatomy and physiology, much like traditional measures of auditory function such as auditory-evoked OAEs, tympanometry and auditory-evoked potentials. Future work will compare subjects with different types of auditory dysfunction to population data from normal hearing subjects. Overall, these findings provide important context for the widespread observations of visual- and eye-movement related signals found in cortical and subcortical auditory areas of the brain.
Collapse
Affiliation(s)
- Cynthia D King
- Department of Neurobiology, Duke University Medical Center, Durham, NC, USA; Department of Psychology and Neuroscience, Duke University, Durham, NC, USA; Center for Cognitive Neuroscience, Duke University, Durham, NC, USA; Duke Institute for Brain Sciences, Duke University, Durham, NC, USA.
| | - Stephanie N Lovich
- Department of Neurobiology, Duke University Medical Center, Durham, NC, USA; Department of Psychology and Neuroscience, Duke University, Durham, NC, USA; Center for Cognitive Neuroscience, Duke University, Durham, NC, USA; Duke Institute for Brain Sciences, Duke University, Durham, NC, USA
| | - David Lk Murphy
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA; Center for Cognitive Neuroscience, Duke University, Durham, NC, USA; Duke Institute for Brain Sciences, Duke University, Durham, NC, USA
| | - Rachel Landrum
- Department of Neurobiology, Duke University Medical Center, Durham, NC, USA; Department of Psychology and Neuroscience, Duke University, Durham, NC, USA; Center for Cognitive Neuroscience, Duke University, Durham, NC, USA; Duke Institute for Brain Sciences, Duke University, Durham, NC, USA
| | - David Kaylie
- Department of Otolaryngology, Duke University Medical Center, Durham, NC, USA
| | - Christopher A Shera
- Department of Otolaryngology, University of Southern California, Los Angeles, CA, USA
| | - Jennifer M Groh
- Department of Neurobiology, Duke University Medical Center, Durham, NC, USA; Department of Psychology and Neuroscience, Duke University, Durham, NC, USA; Center for Cognitive Neuroscience, Duke University, Durham, NC, USA; Duke Institute for Brain Sciences, Duke University, Durham, NC, USA; Department of Computer Science, Duke University, Durham, NC, USA; Department of Biomedical Engineering, Duke University, Durham, NC, USA
| |
Collapse
|
5
|
Lovich SN, King CD, Murphy DLK, Landrum RE, Shera CA, Groh JM. Parametric information about eye movements is sent to the ears. Proc Natl Acad Sci U S A 2023; 120:e2303562120. [PMID: 37988462 PMCID: PMC10691342 DOI: 10.1073/pnas.2303562120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Accepted: 09/28/2023] [Indexed: 11/23/2023] Open
Abstract
Eye movements alter the relationship between the visual and auditory spatial scenes. Signals related to eye movements affect neural pathways from the ear through auditory cortex and beyond, but how these signals contribute to computing the locations of sounds with respect to the visual scene is poorly understood. Here, we evaluated the information contained in eye movement-related eardrum oscillations (EMREOs), pressure changes recorded in the ear canal that occur in conjunction with simultaneous eye movements. We show that EMREOs contain parametric information about horizontal and vertical eye displacement as well as initial/final eye position with respect to the head. The parametric information in the horizontal and vertical directions can be modeled as combining linearly, allowing accurate prediction of the EMREOs associated with oblique (diagonal) eye movements. Target location can also be inferred from the EMREO signals recorded during eye movements to those targets. We hypothesize that the (currently unknown) mechanism underlying EMREOs could impose a two-dimensional eye-movement-related transfer function on any incoming sound, permitting subsequent processing stages to compute the positions of sounds in relation to the visual scene.
Collapse
Affiliation(s)
- Stephanie N. Lovich
- Department of Psychology and Neuroscience, Duke University, Durham, NC27708
- Department of Neurobiology, Duke University, Durham, NC27710
- Center for Cognitive Neuroscience, Duke University, Durham, NC27708
- Duke Institute for Brain Sciences, Duke University, Durham, NC27708
| | - Cynthia D. King
- Department of Psychology and Neuroscience, Duke University, Durham, NC27708
- Department of Neurobiology, Duke University, Durham, NC27710
- Center for Cognitive Neuroscience, Duke University, Durham, NC27708
- Duke Institute for Brain Sciences, Duke University, Durham, NC27708
| | - David L. K. Murphy
- Department of Psychology and Neuroscience, Duke University, Durham, NC27708
- Center for Cognitive Neuroscience, Duke University, Durham, NC27708
- Duke Institute for Brain Sciences, Duke University, Durham, NC27708
| | - Rachel E. Landrum
- Department of Psychology and Neuroscience, Duke University, Durham, NC27708
- Department of Neurobiology, Duke University, Durham, NC27710
- Center for Cognitive Neuroscience, Duke University, Durham, NC27708
- Duke Institute for Brain Sciences, Duke University, Durham, NC27708
| | - Christopher A. Shera
- Department of Otolaryngology, University of Southern California, Los Angeles, CA90007
| | - Jennifer M. Groh
- Department of Psychology and Neuroscience, Duke University, Durham, NC27708
- Department of Neurobiology, Duke University, Durham, NC27710
- Center for Cognitive Neuroscience, Duke University, Durham, NC27708
- Duke Institute for Brain Sciences, Duke University, Durham, NC27708
- Department of Computer Science, Duke University, Durham, NC27708
- Department of Biomedical Engineering, Duke University, Durham, NC27708
| |
Collapse
|
6
|
Nentwich M, Leszczynski M, Russ BE, Hirsch L, Markowitz N, Sapru K, Schroeder CE, Mehta AD, Bickel S, Parra LC. Semantic novelty modulates neural responses to visual change across the human brain. Nat Commun 2023; 14:2910. [PMID: 37217478 PMCID: PMC10203305 DOI: 10.1038/s41467-023-38576-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2022] [Accepted: 05/08/2023] [Indexed: 05/24/2023] Open
Abstract
Our continuous visual experience in daily life is dominated by change. Previous research has focused on visual change due to stimulus motion, eye movements or unfolding events, but not their combined impact across the brain, or their interactions with semantic novelty. We investigate the neural responses to these sources of novelty during film viewing. We analyzed intracranial recordings in humans across 6328 electrodes from 23 individuals. Responses associated with saccades and film cuts were dominant across the entire brain. Film cuts at semantic event boundaries were particularly effective in the temporal and medial temporal lobe. Saccades to visual targets with high visual novelty were also associated with strong neural responses. Specific locations in higher-order association areas showed selectivity to either high or low-novelty saccades. We conclude that neural activity associated with film cuts and eye movements is widespread across the brain and is modulated by semantic novelty.
Collapse
Affiliation(s)
- Maximilian Nentwich
- Department of Biomedical Engineering, The City College of New York, New York, NY, USA.
| | - Marcin Leszczynski
- Departments of Psychiatry and Neurology, Columbia University College of Physicians and Surgeons, New York, NY, USA
- Translational Neuroscience Lab Division, Center for Biomedical Imaging and Neuromodulation, Nathan Kline Institute, Orangeburg, NY, USA
- Cognitive Science Department, Institute of Philosophy, Jagiellonian University, Kraków, Poland
| | - Brian E Russ
- Translational Neuroscience Lab Division, Center for Biomedical Imaging and Neuromodulation, Nathan Kline Institute, Orangeburg, NY, USA
- Nash Family Department of Neuroscience and Friedman Brain Institute, Icahn School of Medicine, New York, NY, USA
- Department of Psychiatry, New York University at Langone, New York, NY, USA
| | - Lukas Hirsch
- Department of Biomedical Engineering, The City College of New York, New York, NY, USA
| | - Noah Markowitz
- The Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
| | - Kaustubh Sapru
- Department of Biomedical Engineering, The City College of New York, New York, NY, USA
| | - Charles E Schroeder
- Departments of Psychiatry and Neurology, Columbia University College of Physicians and Surgeons, New York, NY, USA
- Translational Neuroscience Lab Division, Center for Biomedical Imaging and Neuromodulation, Nathan Kline Institute, Orangeburg, NY, USA
| | - Ashesh D Mehta
- The Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
- Departments of Neurosurgery and Neurology, Zucker School of Medicine at Hofstra/Northwell, Manhasset, NY, USA
| | - Stephan Bickel
- Translational Neuroscience Lab Division, Center for Biomedical Imaging and Neuromodulation, Nathan Kline Institute, Orangeburg, NY, USA
- The Feinstein Institutes for Medical Research, Northwell Health, Manhasset, NY, USA
- Departments of Neurosurgery and Neurology, Zucker School of Medicine at Hofstra/Northwell, Manhasset, NY, USA
| | - Lucas C Parra
- Department of Biomedical Engineering, The City College of New York, New York, NY, USA.
| |
Collapse
|
7
|
Barranco C. Saccades can modulate auditory brain networks. Nat Rev Neurosci 2023; 24:331. [PMID: 37015995 DOI: 10.1038/s41583-023-00700-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/06/2023]
|