1
|
Hansmann D, Derrick D, Theys C. Hearing, seeing, and feeling speech: the neurophysiological correlates of trimodal speech perception. Front Hum Neurosci 2023; 17:1225976. [PMID: 37706173 PMCID: PMC10495990 DOI: 10.3389/fnhum.2023.1225976] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2023] [Accepted: 08/08/2023] [Indexed: 09/15/2023] Open
Abstract
Introduction To perceive speech, our brains process information from different sensory modalities. Previous electroencephalography (EEG) research has established that audio-visual information provides an advantage compared to auditory-only information during early auditory processing. In addition, behavioral research showed that auditory speech perception is not only enhanced by visual information but also by tactile information, transmitted by puffs of air arriving at the skin and aligned with speech. The current EEG study aimed to investigate whether the behavioral benefits of bimodal audio-aerotactile and trimodal audio-visual-aerotactile speech presentation are reflected in cortical auditory event-related neurophysiological responses. Methods To examine the influence of multimodal information on speech perception, 20 listeners conducted a two-alternative forced-choice syllable identification task at three different signal-to-noise levels. Results Behavioral results showed increased syllable identification accuracy when auditory information was complemented with visual information, but did not show the same effect for the addition of tactile information. Similarly, EEG results showed an amplitude suppression for the auditory N1 and P2 event-related potentials for the audio-visual and audio-visual-aerotactile modalities compared to auditory and audio-aerotactile presentations of the syllable/pa/. No statistically significant difference was present between audio-aerotactile and auditory-only modalities. Discussion Current findings are consistent with past EEG research showing a visually induced amplitude suppression during early auditory processing. In addition, the significant neurophysiological effect of audio-visual but not audio-aerotactile presentation is in line with the large benefit of visual information but comparatively much smaller effect of aerotactile information on auditory speech perception previously identified in behavioral research.
Collapse
Affiliation(s)
- Doreen Hansmann
- School of Psychology, Speech and Hearing, University of Canterbury, Christchurch, New Zealand
| | - Donald Derrick
- New Zealand Institute of Language, Brain and Behaviour, University of Canterbury, Christchurch, New Zealand
| | - Catherine Theys
- School of Psychology, Speech and Hearing, University of Canterbury, Christchurch, New Zealand
- New Zealand Institute of Language, Brain and Behaviour, University of Canterbury, Christchurch, New Zealand
| |
Collapse
|
2
|
Ghaneirad E, Saenger E, Szycik GR, Čuš A, Möde L, Sinke C, Wiswede D, Bleich S, Borgolte A. Deficient Audiovisual Speech Perception in Schizophrenia: An ERP Study. Brain Sci 2023; 13:970. [PMID: 37371448 DOI: 10.3390/brainsci13060970] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Revised: 06/15/2023] [Accepted: 06/16/2023] [Indexed: 06/29/2023] Open
Abstract
In everyday verbal communication, auditory speech perception is often disturbed by background noise. Especially in disadvantageous hearing conditions, additional visual articulatory information (e.g., lip movement) can positively contribute to speech comprehension. Patients with schizophrenia (SZs) demonstrate an aberrant ability to integrate visual and auditory sensory input during speech perception. Current findings about underlying neural mechanisms of this deficit are inconsistent. Particularly and despite the importance of early sensory processing in speech perception, very few studies have addressed these processes in SZs. Thus, in the present study, we examined 20 adult subjects with SZ and 21 healthy controls (HCs) while presenting audiovisual spoken words (disyllabic nouns) either superimposed by white noise (-12 dB signal-to-noise ratio) or not. In addition to behavioral data, event-related brain potentials (ERPs) were recorded. Our results demonstrate reduced speech comprehension for SZs compared to HCs under noisy conditions. Moreover, we found altered N1 amplitudes in SZ during speech perception, while P2 amplitudes and the N1-P2 complex were similar to HCs, indicating that there may be disturbances in multimodal speech perception at an early stage of processing, which may be due to deficits in auditory speech perception. Moreover, a positive relationship between fronto-central N1 amplitudes and the positive subscale of the Positive and Negative Syndrome Scale (PANSS) has been observed.
Collapse
Affiliation(s)
- Erfan Ghaneirad
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, 30635 Hanover, Germany
| | - Ellyn Saenger
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, 30635 Hanover, Germany
| | - Gregor R Szycik
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, 30635 Hanover, Germany
| | - Anja Čuš
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, 30635 Hanover, Germany
| | - Laura Möde
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, 30635 Hanover, Germany
| | - Christopher Sinke
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, 30635 Hanover, Germany
| | - Daniel Wiswede
- Department of Neurology, University of Lübeck, 23562 Lübeck, Germany
| | - Stefan Bleich
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, 30635 Hanover, Germany
- Center for Systems Neuroscience, University of Veterinary Medicine, 30559 Hanover, Germany
| | - Anna Borgolte
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, 30635 Hanover, Germany
| |
Collapse
|
3
|
Choi JH, Choi HJ, Kim DH, Park JH, An YH, Shim HJ. Effect of face masks on speech perception in noise of individuals with hearing aids. Front Neurosci 2022; 16:1036767. [PMID: 36532290 PMCID: PMC9754666 DOI: 10.3389/fnins.2022.1036767] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2022] [Accepted: 11/11/2022] [Indexed: 11/03/2023] Open
Abstract
Although several previous studies have confirmed that listeners find it difficult to perceive the speech of face-mask-wearing speakers, there has been little research into how masks affect hearing-impaired individuals using hearing aids. Therefore, the aim of this study was to compare the effects of masks on the speech perception in noise of hearing-impaired individuals and normal-hearing individuals. We also investigated the effect of masks on the gain conferred by hearing aids. The hearing-impaired group included 24 listeners (age: M = 69.5, SD = 8.6; M:F = 13:11) who had used hearing aids in everyday life for >1 month (M = 20.7, SD = 24.0) and the normal-hearing group included 26 listeners (age: M = 57.9, SD = 11.1; M:F = 13:13). Speech perception in noise was measured under no mask-auditory-only (no-mask-AO), no mask-auditory-visual (no-mask-AV), and mask-AV conditions at five signal-to-noise ratios (SNRs; -16, -12, -8, -4, 0 dB) using five lists of 25 monosyllabic Korean words. Video clips that included a female speaker's face and sound or the sound only were presented through a monitor and a loudspeaker located 1 m in front of the listener in a sound-attenuating booth. The degree of deterioration in speech perception caused by the mask (no-mask-AV minus mask-AV) was significantly greater for hearing-impaired vs. normal-hearing participants only at 0 dB SNR (Bonferroni's corrected p < 0.01). When the effects of a mask on speech perception, with and without hearing aids, were compared in the hearing-impaired group, the degree of deterioration in speech perception caused by the mask was significantly reduced by the hearing aids compared with that without hearing aids at 0 and -4 dB SNR (Bonferroni's corrected p < 0.01). The improvement conferred by hearing aids (unaided speech perception score minus aided speech perception score) was significantly greater at 0 and -4 dB SNR than at -16 dB SNR in the mask-AV group (Bonferroni's corrected p < 0.01). These results demonstrate that hearing aids still improve speech perception when the speaker is masked, and that hearing aids partly offset the effect of a mask at relatively low noise levels.
Collapse
Affiliation(s)
| | | | | | | | | | - Hyun Joon Shim
- Department of Otorhinolaryngology-Head and Neck Surgery, Nowon Eulji Medical Center, Eulji University School of Medicine, Seoul, South Korea
| |
Collapse
|
4
|
Borgolte A, Roy M, Sinke C, Wiswede D, Stephan M, Bleich S, Münte TF, Szycik GR. Enhanced attentional processing during speech perception in adult high-functioning autism spectrum disorder: An ERP-study. Neuropsychologia 2021; 161:108022. [PMID: 34530026 DOI: 10.1016/j.neuropsychologia.2021.108022] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2020] [Revised: 08/17/2021] [Accepted: 09/09/2021] [Indexed: 12/16/2022]
Abstract
Deficits in audiovisual speech perception have consistently been detected in patients with Autism Spectrum Disorder (ASD). Especially for patients with a highly functional subtype of ASD, it remains uncertain whether these deficits and underlying neural mechanisms persist into adulthood. Research indicates differences in audiovisual speech processing between ASD and healthy controls (HC) in the auditory cortex. The temporal dynamics of these differences still need to be characterized. Thus, in the present study we examined 14 adult subjects with high-functioning ASD and 15 adult HC while they viewed visual (lip movements) and auditory (voice) speech information that was either superimposed by white noise (condition 1) or not (condition 2). Subject's performance was quantified by measuring stimulus comprehension. In addition, event-related brain potentials (ERPs) were recorded. Results demonstrated worse speech comprehension for ASD subjects compared to HC under noisy conditions. Moreover, ERP-analysis revealed significantly higher P2 amplitudes over parietal electrodes for ASD subjects compared to HC.
Collapse
Affiliation(s)
- Anna Borgolte
- Dept. of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany.
| | - Mandy Roy
- Asklepios, Psychiatric Hospital Ochsenzoll, Hamburg, Germany
| | - Christopher Sinke
- Dept. of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany
| | - Daniel Wiswede
- Dept. of Neurology, University of Lübeck, Lübeck, Germany
| | - Michael Stephan
- Dept. of Psychosomatic Medicine and Psychotherapy, Hannover Medical School, Hanover, Germany
| | - Stefan Bleich
- Dept. of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany; Center of Systems Neuroscience, Hanover, Germany
| | - Thomas F Münte
- Dept. of Neurology, University of Lübeck, Lübeck, Germany
| | - Gregor R Szycik
- Dept. of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany
| |
Collapse
|
5
|
Yuan Y, Meyers K, Borges K, Lleo Y, Fiorentino KA, Oh Y. Effects of Visual Speech Envelope on Audiovisual Speech Perception in Multitalker Listening Environments. JOURNAL OF SPEECH, LANGUAGE, AND HEARING RESEARCH : JSLHR 2021; 64:2845-2853. [PMID: 34100628 DOI: 10.1044/2021_jslhr-20-00688] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Purpose This study investigated the effects of visually presented speech envelope information with various modulation rates and depths on audiovisual speech perception in noise. Method Forty adults (21.25 ± 1.45 years) participated in audiovisual sentence recognition measurements in noise. Target speech sentences were auditorily presented in multitalker babble noises at a -3 dB SNR. Acoustic amplitude envelopes of target signals were extracted through low-pass filters with different cutoff frequencies (4, 10, and 30 Hz) and a fixed modulation depth at 100% (Experiment 1) or extracted with various modulation depths (0%, 25%, 50%, 75%, and 100%) and a fixed 10-Hz modulation rate (Experiment 2). The extracted target envelopes were synchronized with the amplitude of a spherical-shaped ball and presented as visual stimuli. Subjects were instructed to attend to both auditory and visual stimuli of the target sentences and type down their answers. The sentence recognition accuracy was compared between audio-only and audiovisual conditions. Results In Experiment 1, a significant improvement in speech intelligibility was observed when the visual analog (a sphere) synced with the acoustic amplitude envelope modulated at a 10-Hz modulation rate compared to the audio-only condition. In Experiment 2, the visual analog with 75% modulation depth resulted in better audiovisual speech perception in noise compared to the other modulation depth conditions. Conclusion An abstract visual analog of acoustic amplitude envelopes can be efficiently delivered by the visual system and integrated online with auditory signals to enhance speech perception in noise, independent of particular articulation movements.
Collapse
Affiliation(s)
- Yi Yuan
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville
| | - Kelli Meyers
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville
| | - Kayla Borges
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville
| | - Yasneli Lleo
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville
| | - Katarina A Fiorentino
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville
| | - Yonghee Oh
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville
| |
Collapse
|
6
|
Yuan Y, Lleo Y, Daniel R, White A, Oh Y. The Impact of Temporally Coherent Visual Cues on Speech Perception in Complex Auditory Environments. Front Neurosci 2021; 15:678029. [PMID: 34163326 PMCID: PMC8216555 DOI: 10.3389/fnins.2021.678029] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2021] [Accepted: 05/04/2021] [Indexed: 11/22/2022] Open
Abstract
Speech perception often takes place in noisy environments, where multiple auditory signals compete with one another. The addition of visual cues such as talkers’ faces or lip movements to an auditory signal can help improve the intelligibility of speech in those suboptimal listening environments. This is referred to as audiovisual benefits. The current study aimed to delineate the signal-to-noise ratio (SNR) conditions under which visual presentations of the acoustic amplitude envelopes have their most significant impact on speech perception. Seventeen adults with normal hearing were recruited. Participants were presented with spoken sentences in babble noise either in auditory-only or auditory-visual conditions with various SNRs at −7, −5, −3, −1, and 1 dB. The visual stimulus applied in this study was a sphere that varied in size syncing with the amplitude envelope of the target speech signals. Participants were asked to transcribe the sentences they heard. Results showed that a significant improvement in accuracy in the auditory-visual condition versus the audio-only condition was obtained at the SNRs of −3 and −1 dB, but no improvement was observed in other SNRs. These results showed that dynamic temporal visual information can benefit speech perception in noise, and the optimal facilitative effects of visual amplitude envelope can be observed under an intermediate SNR range.
Collapse
Affiliation(s)
- Yi Yuan
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville, FL, United States
| | - Yasneli Lleo
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville, FL, United States
| | - Rebecca Daniel
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville, FL, United States
| | - Alexandra White
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville, FL, United States
| | - Yonghee Oh
- Department of Speech, Language, and Hearing Sciences, University of Florida, Gainesville, FL, United States
| |
Collapse
|
7
|
Lu L, Liu B. Revealing the multisensory modulation of auditory stimulus in degraded visual object recognition by dynamic causal modeling. Brain Imaging Behav 2019; 14:1187-1198. [PMID: 31172360 DOI: 10.1007/s11682-019-00134-3] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Recent evidence from neurophysiological and functional imaging research has demonstrated that semantically congruent sounds can modulate the identification of a degraded visual object. However, it remains unclear how different integration regions interact with each other when only a visual object was obscured. The present study aimed to elucidate the neural bases of cross-modal functional interactions in degraded visual object recognition. Naturally degraded images and semantically congruent sounds were used in our experiment. Participants were presented with three different modalities of audio-visual stimuli: auditory only (A), degraded visual only (Vd), and simultaneous auditory and degraded visual (AVd). We used conjunction analysis and the classical 'max criterion' to define three audiovisual integration cortical hubs: the visual association cortex, the superior temporal sulcus and the Heschl's gyrus. Dynamic causal modeling (DCM) was then used to infer effective connectivity between these regions. The DCM results revealed that the modulation of an auditory stimulus resulted in increased connectivity from the Heschl's gyrus to the visual association cortex and from the superior temporal sulcus to the visual association cortex. It was shown that the visual association cortex is modulated not only via feedback and top-down connections from higher-order convergence areas but also via lateral feedforward connectivity from the auditory cortex. The present findings give support to interconnected models of cross-modal information integration.
Collapse
Affiliation(s)
- Lu Lu
- Institute of Disaster Medicine, Tianjin University, Tianjin, 300072, People's Republic of China
| | - Baolin Liu
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, People's Republic of China.
| |
Collapse
|
8
|
Lu L, Zhang G, Xu J, Liu B. Semantically Congruent Sounds Facilitate the Decoding of Degraded Images. Neuroscience 2018; 377:12-25. [PMID: 29408368 DOI: 10.1016/j.neuroscience.2018.01.051] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2017] [Revised: 01/20/2018] [Accepted: 01/23/2018] [Indexed: 11/19/2022]
Abstract
Semantically congruent sounds can facilitate perception of visual objects in the human brain. However, the manner in which semantically congruent sounds affect cognitive processing for degraded visual stimuli remains unclear. We presented participants with naturalistic degraded images and semantically congruent sounds from different conceptual categories in three modalities: degraded visual only, auditory only, and auditory and degraded visual. Functional magnetic resonance imaging was performed to assess variations in brain-activation spatial patterns. In order to account for the facilitation of auditory modulation at different levels, four conceptual categories of stimuli were divided into coarse and fine groups. Conjunction analysis and multivariate pattern analysis were used to investigate integrative properties. Superadditive interactions were found in the visual association cortex and subadditive interactions were observed in the superior temporal sulcus/superior temporal gyrus (STS/STG). Our results demonstrate that the visual association cortex and STS/STG are involved in the integration of auditory and degraded visual information. In addition, the pattern classification results imply that semantically congruent sounds may facilitate identification of degraded images in both coarse and fine groups. Importantly, when naturalistic visual stimuli were further subdivided, facilitation through auditory modulation exhibited category selectivity.
Collapse
Affiliation(s)
- Lu Lu
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300350, PR China
| | - Gaoyan Zhang
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300350, PR China
| | - Junhai Xu
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300350, PR China
| | - Baolin Liu
- School of Computer Science and Technology, Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin 300350, PR China; State Key Laboratory of Intelligent Technology and Systems, National Laboratory for Information Science and Technology, Tsinghua University, Beijing 100084, PR China.
| |
Collapse
|
9
|
Baart M. Quantifying lip-read-induced suppression and facilitation of the auditory N1 and P2 reveals peak enhancements and delays. Psychophysiology 2016; 53:1295-306. [DOI: 10.1111/psyp.12683] [Citation(s) in RCA: 32] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2015] [Accepted: 05/09/2016] [Indexed: 11/29/2022]
Affiliation(s)
- Martijn Baart
- BCBL. Basque Center on Cognition, Brain and Language; Donostia-San Sebastián Spain
- Department of Cognitive Neuropsychology; Tilburg University; Tilburg The Netherlands
| |
Collapse
|
10
|
Krueger Fister J, Stevenson RA, Nidiffer AR, Barnett ZP, Wallace MT. Stimulus intensity modulates multisensory temporal processing. Neuropsychologia 2016; 88:92-100. [PMID: 26920937 DOI: 10.1016/j.neuropsychologia.2016.02.016] [Citation(s) in RCA: 38] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2015] [Revised: 01/20/2016] [Accepted: 02/22/2016] [Indexed: 12/18/2022]
Abstract
One of the more challenging feats that multisensory systems must perform is to determine which sensory signals originate from the same external event, and thus should be integrated or "bound" into a singular perceptual object or event, and which signals should be segregated. Two important stimulus properties impacting this process are the timing and effectiveness of the paired stimuli. It has been well established that the more temporally aligned two stimuli are, the greater the degree to which they influence one another's processing. In addition, the less effective the individual unisensory stimuli are in eliciting a response, the greater the benefit when they are combined. However, the interaction between stimulus timing and stimulus effectiveness in driving multisensory-mediated behaviors has never been explored - which was the purpose of the current study. Participants were presented with either high- or low-intensity audiovisual stimuli in which stimulus onset asynchronies (SOAs) were parametrically varied, and were asked to report on the perceived synchrony/asynchrony of the paired stimuli. Our results revealed an interaction between the temporal relationship (SOA) and intensity of the stimuli. Specifically, individuals were more tolerant of larger temporal offsets (i.e., more likely to call them synchronous) when the paired stimuli were less effective. This interaction was also seen in response time (RT) distributions. Behavioral gains in RTs were seen with synchronous relative to asynchronous presentations, but this effect was more pronounced with high-intensity stimuli. These data suggest that stimulus effectiveness plays an underappreciated role in the perception of the timing of multisensory events, and reinforces the interdependency of the principles of multisensory integration in determining behavior and shaping perception.
Collapse
Affiliation(s)
- Juliane Krueger Fister
- Neuroscience Graduate Program, Vanderbilt University Medical Center, United States; Vanderbilt Brain Institute, United States.
| | - Ryan A Stevenson
- Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, United States; Vanderbilt Brain Institute, United States; Vanderbilt University Kennedy Center, United States; Department of Psychology, University of Toronto, Canada
| | - Aaron R Nidiffer
- Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, United States
| | - Zachary P Barnett
- Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, United States
| | - Mark T Wallace
- Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, United States; Vanderbilt Brain Institute, United States; Vanderbilt University Kennedy Center, United States; Department of Psychology, Vanderbilt University, United States; Department of Psychiatry, Vanderbilt University, United States
| |
Collapse
|
11
|
Méndez-Balbuena I, Huidobro N, Silva M, Flores A, Trenado C, Quintanar L, Arias-Carrión O, Kristeva R, Manjarrez E. Effect of mechanical tactile noise on amplitude of visual evoked potentials: multisensory stochastic resonance. J Neurophysiol 2015; 114:2132-43. [PMID: 26156387 DOI: 10.1152/jn.00457.2015] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2015] [Accepted: 07/06/2015] [Indexed: 11/22/2022] Open
Abstract
The present investigation documents the electrophysiological occurrence of multisensory stochastic resonance in the human visual pathway elicited by tactile noise. We define multisensory stochastic resonance of brain evoked potentials as the phenomenon in which an intermediate level of input noise of one sensory modality enhances the brain evoked response of another sensory modality. Here we examined this phenomenon in visual evoked potentials (VEPs) modulated by the addition of tactile noise. Specifically, we examined whether a particular level of mechanical Gaussian noise applied to the index finger can improve the amplitude of the VEP. We compared the amplitude of the positive P100 VEP component between zero noise (ZN), optimal noise (ON), and high mechanical noise (HN). The data disclosed an inverted U-like graph for all the subjects, thus demonstrating the occurrence of a multisensory stochastic resonance in the P100 VEP.
Collapse
Affiliation(s)
| | - Nayeli Huidobro
- Instituto de Fisiología, Benemérita Universidad Autónoma de Puebla, Puebla, Mexico
| | - Mayte Silva
- Instituto de Fisiología, Benemérita Universidad Autónoma de Puebla, Puebla, Mexico
| | - Amira Flores
- Instituto de Fisiología, Benemérita Universidad Autónoma de Puebla, Puebla, Mexico
| | - Carlos Trenado
- Institute of Clinical Neuroscience, Heinrich Heine University, Düsseldorf, Germany
| | - Luis Quintanar
- Facultad de Psicología, Benemérita Universidad Autónoma de Puebla, Puebla, Mexico
| | - Oscar Arias-Carrión
- Unidad de Trastornos del Movimiento y Sueño (TMS), Hospital General Dr. Manuel Gea González/IFC-UNAM, Mexico City, Mexico; and
| | - Rumyana Kristeva
- Department of Neurology, University of Freiburg, Freiburg, Germany
| | - Elias Manjarrez
- Instituto de Fisiología, Benemérita Universidad Autónoma de Puebla, Puebla, Mexico;
| |
Collapse
|
12
|
Lin Y, Liu B, Liu Z, Gao X. EEG gamma-band activity during audiovisual speech comprehension in different noise environments. Cogn Neurodyn 2015; 9:389-98. [PMID: 26157512 DOI: 10.1007/s11571-015-9333-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2014] [Revised: 01/15/2015] [Accepted: 01/23/2015] [Indexed: 10/24/2022] Open
Abstract
The presence of cross-modal stochastic resonance in different noise environments has been proved in previous behavioral and event-related potential studies, while it was still unclear whether the gamma-band oscillation study was another evidence of cross-modal stochastic resonance. The multisensory gain of gamma-band activity between the audiovisual (AV) and auditory-only conditions in different noise environments was analyzed. Videos of face motion articulating words concordant with different levels of pink noise were used as stimuli. Signal-to-noise ratios (SNRs) of 0, -4, -8, -12 and -16 dB were selected to measure the speech recognition accuracy and EEG activity for 20 healthy subjects. The power and phase of EEG gamma-band oscillations increased in a time window of 50-90 ms. The multisensory gains of evoked and total activity, as well as phase-locking factor, were greatest at the -12 dB SNR, which were consistent with the behavioral result. The multisensory gain of gamma-band activity showed an inverted U-shaped curve as a function of SNR. This finding confirmed the presence of cross-modal stochastic resonance. In addition, there was a significant correlation between evoked activity and phase-locking factor of gamma-band at five different SNRs. Gamma-band oscillation was believed to play a role in the rapid processing and information linkage strengthening of AV modalities in the early stage of cognitive processes.
Collapse
Affiliation(s)
- Yanfei Lin
- School of Medicine, Tsinghua University, Beijing, 100084 People's Republic of China ; School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of China
| | - Baolin Liu
- School of Computer Science and Technology, Tianjin University, Tianjin, 300072 People's Republic of China
| | - Zhiwen Liu
- School of Information and Electronics, Beijing Institute of Technology, Beijing, 100081 People's Republic of China
| | - Xiaorong Gao
- School of Medicine, Tsinghua University, Beijing, 100084 People's Republic of China
| |
Collapse
|
13
|
Altieri N, Wenger MJ. Neural dynamics of audiovisual speech integration under variable listening conditions: an individual participant analysis. Front Psychol 2013; 4:615. [PMID: 24058358 PMCID: PMC3767908 DOI: 10.3389/fpsyg.2013.00615] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2013] [Accepted: 08/22/2013] [Indexed: 11/25/2022] Open
Abstract
Speech perception engages both auditory and visual modalities. Limitations of traditional accuracy-only approaches in the investigation of audiovisual speech perception have motivated the use of new methodologies. In an audiovisual speech identification task, we utilized capacity (Townsend and Nozawa, 1995), a dynamic measure of efficiency, to quantify audiovisual integration. Capacity was used to compare RT distributions from audiovisual trials to RT distributions from auditory-only and visual-only trials across three listening conditions: clear auditory signal, S/N ratio of −12 dB, and S/N ratio of −18 dB. The purpose was to obtain EEG recordings in conjunction with capacity to investigate how a late ERP co-varies with integration efficiency. Results showed efficient audiovisual integration for low auditory S/N ratios, but inefficient audiovisual integration when the auditory signal was clear. The ERP analyses showed evidence for greater audiovisual amplitude compared to the unisensory signals for lower auditory S/N ratios (higher capacity/efficiency) compared to the high S/N ratio (low capacity/inefficient integration). The data are consistent with an interactive framework of integration, where auditory recognition is influenced by speech-reading as a function of signal clarity.
Collapse
Affiliation(s)
- Nicholas Altieri
- Department of Communication Sciences and Disorders, Idaho State University Pocatello, ID, USA
| | | |
Collapse
|