1
|
Ricou C, Rabadan V, Mofid Y, Aguillon-Hernandez N, Wardak C. Pupil dilation reflects the social and motion content of faces. Soc Cogn Affect Neurosci 2024; 19:nsae055. [PMID: 39167473 PMCID: PMC11403811 DOI: 10.1093/scan/nsae055] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Revised: 07/15/2024] [Accepted: 08/19/2024] [Indexed: 08/23/2024] Open
Abstract
Human facial features (eyes, nose, and mouth) allow us to communicate with others. Observing faces triggers physiological responses, including pupil dilation. Still, the relative influence of social and motion content of a visual stimulus on pupillary reactivity has never been elucidated. A total of 30 adults aged 18-33 years old were recorded with an eye tracker. We analysed the event-related pupil dilation in response to stimuli distributed along a gradient of social salience (non-social to social, going from objects to avatars to real faces) and dynamism (static to micro- to macro-motion). Pupil dilation was larger in response to social (faces and avatars) compared to non-social stimuli (objects), with surprisingly a larger response for avatars. Pupil dilation was also larger in response to macro-motion compared to static. After quantifying each stimulus' real quantity of motion, we found that the higher the quantity of motion, the larger the pupil dilated. However, the slope of this relationship was not higher for social stimuli. Overall, pupil dilation was more sensitive to the real quantity of motion than to the social component of motion, highlighting the relevance of ecological stimulations. Physiological response to faces results from specific contributions of both motion and social processing.
Collapse
Affiliation(s)
- Camille Ricou
- Université de Tours, INSERM, Imaging Brain & Neuropsychiatry iBraiN U1253, Tours 37032, France
| | - Vivien Rabadan
- Université de Tours, INSERM, Imaging Brain & Neuropsychiatry iBraiN U1253, Tours 37032, France
| | - Yassine Mofid
- Université de Tours, INSERM, Imaging Brain & Neuropsychiatry iBraiN U1253, Tours 37032, France
| | | | - Claire Wardak
- Université de Tours, INSERM, Imaging Brain & Neuropsychiatry iBraiN U1253, Tours 37032, France
| |
Collapse
|
2
|
Hauw F, Béranger B, Cohen L. Subtitled speech: the neural mechanisms of ticker-tape synaesthesia. Brain 2024; 147:2530-2541. [PMID: 38620012 PMCID: PMC11224615 DOI: 10.1093/brain/awae114] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Revised: 02/21/2024] [Accepted: 03/21/2024] [Indexed: 04/17/2024] Open
Abstract
The acquisition of reading modifies areas of the brain associated with vision and with language, in addition to their connections. These changes enable reciprocal translation between orthography and the sounds and meaning of words. Individual variability in the pre-existing cerebral substrate contributes to the range of eventual reading abilities, extending to atypical developmental patterns, including dyslexia and reading-related synaesthesias. The present study is devoted to the little-studied but highly informative ticker-tape synaesthesia, in which speech perception triggers the vivid and irrepressible perception of words in their written form in the mind's eye. We scanned a group of 17 synaesthetes and 17 matched controls with functional MRI, while they listened to spoken sentences, words, numbers or pseudowords (Experiment 1), viewed images and written words (Experiment 2) or were at rest (Experiment 3). First, we found direct correlates of the ticker-tape synaesthesia phenomenon: during speech perception, as ticker-tape synaesthesia was active, synaesthetes showed over-activation of left perisylvian regions supporting phonology and of the occipitotemporal visual word form area, where orthography is represented. Second, we provided support to the hypothesis that ticker-tape synaesthesia results from atypical relationships between spoken and written language processing: the ticker-tape synaesthesia-related regions overlap closely with cortices activated during reading, and the overlap of speech-related and reading-related areas is larger in synaesthetes than in controls. Furthermore, the regions over-activated in ticker-tape synaesthesia overlap with regions under-activated in dyslexia. Third, during the resting state (i.e. in the absence of current ticker-tape synaesthesia), synaesthetes showed increased functional connectivity between left prefrontal and bilateral occipital regions. This pattern might reflect a lowered threshold for conscious access to visual mental contents and might imply a non-specific predisposition to all synaesthesias with a visual content. These data provide a rich and coherent account of ticker-tape synaesthesia as a non-detrimental developmental condition created by the interaction of reading acquisition with an atypical cerebral substrate.
Collapse
Affiliation(s)
- Fabien Hauw
- Inserm U 1127, CNRS UMR 7225, Sorbonne Universités, Institut du Cerveau, ICM, Paris 75013, France
- AP-HP, Hôpital de La Pitié Salpêtrière, Fédération de Neurologie, Paris 75013, France
| | - Benoît Béranger
- Inserm U 1127, CNRS UMR 7225, Sorbonne Universités, Institut du Cerveau, ICM, Paris 75013, France
| | - Laurent Cohen
- Inserm U 1127, CNRS UMR 7225, Sorbonne Universités, Institut du Cerveau, ICM, Paris 75013, France
- AP-HP, Hôpital de La Pitié Salpêtrière, Fédération de Neurologie, Paris 75013, France
| |
Collapse
|
3
|
Krason A, Vigliocco G, Mailend ML, Stoll H, Varley R, Buxbaum LJ. Benefit of visual speech information for word comprehension in post-stroke aphasia. Cortex 2023; 165:86-100. [PMID: 37271014 PMCID: PMC10850036 DOI: 10.1016/j.cortex.2023.04.011] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2022] [Revised: 03/13/2023] [Accepted: 04/22/2023] [Indexed: 06/06/2023]
Abstract
Aphasia is a language disorder that often involves speech comprehension impairments affecting communication. In face-to-face settings, speech is accompanied by mouth and facial movements, but little is known about the extent to which they benefit aphasic comprehension. This study investigated the benefit of visual information accompanying speech for word comprehension in people with aphasia (PWA) and the neuroanatomic substrates of any benefit. Thirty-six PWA and 13 neurotypical matched control participants performed a picture-word verification task in which they indicated whether a picture of an animate/inanimate object matched a subsequent word produced by an actress in a video. Stimuli were either audiovisual (with visible mouth and facial movements) or auditory-only (still picture of a silhouette) with audio being clear (unedited) or degraded (6-band noise-vocoding). We found that visual speech information was more beneficial for neurotypical participants than PWA, and more beneficial for both groups when speech was degraded. A multivariate lesion-symptom mapping analysis for the degraded speech condition showed that lesions to superior temporal gyrus, underlying insula, primary and secondary somatosensory cortices, and inferior frontal gyrus were associated with reduced benefit of audiovisual compared to auditory-only speech, suggesting that the integrity of these fronto-temporo-parietal regions may facilitate cross-modal mapping. These findings provide initial insights into our understanding of the impact of audiovisual information on comprehension in aphasia and the brain regions mediating any benefit.
Collapse
Affiliation(s)
- Anna Krason
- Experimental Psychology, University College London, UK; Moss Rehabilitation Research Institute, Elkins Park, PA, USA.
| | - Gabriella Vigliocco
- Experimental Psychology, University College London, UK; Moss Rehabilitation Research Institute, Elkins Park, PA, USA
| | - Marja-Liisa Mailend
- Moss Rehabilitation Research Institute, Elkins Park, PA, USA; Department of Special Education, University of Tartu, Tartu Linn, Estonia
| | - Harrison Stoll
- Moss Rehabilitation Research Institute, Elkins Park, PA, USA; Applied Cognitive and Brain Science, Drexel University, Philadelphia, PA, USA
| | | | - Laurel J Buxbaum
- Moss Rehabilitation Research Institute, Elkins Park, PA, USA; Department of Rehabilitation Medicine, Thomas Jefferson University, Philadelphia, PA, USA
| |
Collapse
|
4
|
Landsiedel J, Koldewyn K. Auditory dyadic interactions through the "eye" of the social brain: How visual is the posterior STS interaction region? IMAGING NEUROSCIENCE (CAMBRIDGE, MASS.) 2023; 1:1-20. [PMID: 37719835 PMCID: PMC10503480 DOI: 10.1162/imag_a_00003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/16/2023] [Accepted: 05/17/2023] [Indexed: 09/19/2023]
Abstract
Human interactions contain potent social cues that meet not only the eye but also the ear. Although research has identified a region in the posterior superior temporal sulcus as being particularly sensitive to visually presented social interactions (SI-pSTS), its response to auditory interactions has not been tested. Here, we used fMRI to explore brain response to auditory interactions, with a focus on temporal regions known to be important in auditory processing and social interaction perception. In Experiment 1, monolingual participants listened to two-speaker conversations (intact or sentence-scrambled) and one-speaker narrations in both a known and an unknown language. Speaker number and conversational coherence were explored in separately localised regions-of-interest (ROI). In Experiment 2, bilingual participants were scanned to explore the role of language comprehension. Combining univariate and multivariate analyses, we found initial evidence for a heteromodal response to social interactions in SI-pSTS. Specifically, right SI-pSTS preferred auditory interactions over control stimuli and represented information about both speaker number and interactive coherence. Bilateral temporal voice areas (TVA) showed a similar, but less specific, profile. Exploratory analyses identified another auditory-interaction sensitive area in anterior STS. Indeed, direct comparison suggests modality specific tuning, with SI-pSTS preferring visual information while aSTS prefers auditory information. Altogether, these results suggest that right SI-pSTS is a heteromodal region that represents information about social interactions in both visual and auditory domains. Future work is needed to clarify the roles of TVA and aSTS in auditory interaction perception and further probe right SI-pSTS interaction-selectivity using non-semantic prosodic cues.
Collapse
Affiliation(s)
- Julia Landsiedel
- Department of Psychology, School of Human and Behavioural Sciences, Bangor University, Bangor, United Kingdom
| | - Kami Koldewyn
- Department of Psychology, School of Human and Behavioural Sciences, Bangor University, Bangor, United Kingdom
| |
Collapse
|
5
|
Hauw F, El Soudany M, Rosso C, Daunizeau J, Cohen L. A single case neuroimaging study of tickertape synesthesia. Sci Rep 2023; 13:12185. [PMID: 37500762 PMCID: PMC10374523 DOI: 10.1038/s41598-023-39276-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2022] [Accepted: 07/22/2023] [Indexed: 07/29/2023] Open
Abstract
Reading acquisition is enabled by deep changes in the brain's visual system and language areas, and in the links subtending their collaboration. Disruption of those plastic processes commonly results in developmental dyslexia. However, atypical development of reading mechanisms may occasionally result in ticker-tape synesthesia (TTS), a condition described by Francis Galton in 1883 wherein individuals "see mentally in print every word that is uttered (…) as from a long imaginary strip of paper". While reading is the bottom-up translation of letters into speech, TTS may be viewed as its opposite, the top-down translation of speech into internally visualized letters. In a series of functional MRI experiments, we studied MK, a man with TTS. We showed that a set of left-hemispheric areas were more active in MK than in controls during the perception of normal than reversed speech, including frontoparietal areas involved in speech processing, and the Visual Word Form Area, an occipitotemporal region subtending orthography. Those areas were identical to those involved in reading, supporting the construal of TTS as upended reading. Using dynamic causal modeling, we further showed that, parallel to reading, TTS induced by spoken words and pseudowords relied on top-down flow of information along distinct lexical and phonological routes, involving the middle temporal and supramarginal gyri, respectively. Future studies of TTS should shed new light on the neurodevelopmental mechanisms of reading acquisition, their variability and their disorders.
Collapse
Affiliation(s)
- Fabien Hauw
- Inserm U 1127, CNRS UMR 7225, Sorbonne Universités, Institut du Cerveau, ICM, Paris, France.
- AP-HP, Hôpital de la Pitié Salpêtrière, Fédération de Neurologie, Paris, France.
| | - Mohamed El Soudany
- Inserm U 1127, CNRS UMR 7225, Sorbonne Universités, Institut du Cerveau, ICM, Paris, France
| | - Charlotte Rosso
- Inserm U 1127, CNRS UMR 7225, Sorbonne Universités, Institut du Cerveau, ICM, Paris, France
- AP-HP, Urgences Cérébro-Vasculaires, Hôpital Pitié-Salpêtrière, Paris, France
| | - Jean Daunizeau
- Inserm U 1127, CNRS UMR 7225, Sorbonne Universités, Institut du Cerveau, ICM, Paris, France
| | - Laurent Cohen
- Inserm U 1127, CNRS UMR 7225, Sorbonne Universités, Institut du Cerveau, ICM, Paris, France
- AP-HP, Hôpital de la Pitié Salpêtrière, Fédération de Neurologie, Paris, France
| |
Collapse
|
6
|
Pepper JL, Nuttall HE. Age-Related Changes to Multisensory Integration and Audiovisual Speech Perception. Brain Sci 2023; 13:1126. [PMID: 37626483 PMCID: PMC10452685 DOI: 10.3390/brainsci13081126] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 07/20/2023] [Accepted: 07/22/2023] [Indexed: 08/27/2023] Open
Abstract
Multisensory integration is essential for the quick and accurate perception of our environment, particularly in everyday tasks like speech perception. Research has highlighted the importance of investigating bottom-up and top-down contributions to multisensory integration and how these change as a function of ageing. Specifically, perceptual factors like the temporal binding window and cognitive factors like attention and inhibition appear to be fundamental in the integration of visual and auditory information-integration that may become less efficient as we age. These factors have been linked to brain areas like the superior temporal sulcus, with neural oscillations in the alpha-band frequency also being implicated in multisensory processing. Age-related changes in multisensory integration may have significant consequences for the well-being of our increasingly ageing population, affecting their ability to communicate with others and safely move through their environment; it is crucial that the evidence surrounding this subject continues to be carefully investigated. This review will discuss research into age-related changes in the perceptual and cognitive mechanisms of multisensory integration and the impact that these changes have on speech perception and fall risk. The role of oscillatory alpha activity is of particular interest, as it may be key in the modulation of multisensory integration.
Collapse
Affiliation(s)
| | - Helen E. Nuttall
- Department of Psychology, Lancaster University, Bailrigg LA1 4YF, UK;
| |
Collapse
|
7
|
Dopierała AAW, Pérez DL, Mercure E, Pluta A, Malinowska-Korczak A, Evans S, Wolak T, Tomalski P. The Development of Cortical Responses to the Integration of Audiovisual Speech in Infancy. Brain Topogr 2023:10.1007/s10548-023-00959-8. [PMID: 37171657 PMCID: PMC10176292 DOI: 10.1007/s10548-023-00959-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2022] [Accepted: 04/11/2023] [Indexed: 05/13/2023]
Abstract
In adults, the integration of audiovisual speech elicits specific higher (super-additive) or lower (sub-additive) cortical responses when compared to the responses to unisensory stimuli. Although there is evidence that the fronto-temporal network is active during perception of audiovisual speech in infancy, the development of fronto-temporal responses to audiovisual integration remains unknown. In the current study, 5-month-olds and 10-month-olds watched bimodal (audiovisual) and alternating unimodal (auditory + visual) syllables. In this context we use alternating unimodal to denote alternating auditory and visual syllables that are perceived as separate syllables by adults. Using fNIRS we measured responses over large cortical areas including the inferior frontal and superior temporal regions. We identified channels showing different responses to bimodal than alternating unimodal condition and used multivariate pattern analysis (MVPA) to decode patterns of cortical responses to bimodal (audiovisual) and alternating unimodal (auditory + visual) speech. Results showed that in both age groups integration elicits cortical responses consistent with both super- and sub-additive responses in the fronto-temporal cortex. The univariate analyses revealed that between 5 and 10 months spatial distribution of these responses becomes increasingly focal. MVPA correctly classified responses at 5 months, with key input from channels located in the inferior frontal and superior temporal channels of the right hemisphere. However, MVPA classification was not successful at 10 months, suggesting a potential cortical re-organisation of audiovisual speech perception at this age. These results show the complex and non-gradual development of the cortical responses to integration of congruent audiovisual speech in infancy.
Collapse
Affiliation(s)
- Aleksandra A W Dopierała
- Faculty of Psychology, University of Warsaw, Warsaw, Poland.
- Department of Psychology, University of British Columbia, Vancouver, Canada.
| | | | | | - Agnieszka Pluta
- Faculty of Psychology, University of Warsaw, Warsaw, Poland
- University of Westminister, London, UK
| | | | - Samuel Evans
- Kings College London, London, UK
- University of Westminister, London, UK
| | - Tomasz Wolak
- Institute of Physiology and Pathology of Hearing, Bioimaging Research Center, World Hearing Centre, Warsaw, Poland
| | - Przemysław Tomalski
- Faculty of Psychology, University of Warsaw, Warsaw, Poland.
- Institute of Psychology, Polish Academy of Sciences, Warsaw, Poland.
| |
Collapse
|
8
|
Scheliga S, Kellermann T, Lampert A, Rolke R, Spehr M, Habel U. Neural correlates of multisensory integration in the human brain: an ALE meta-analysis. Rev Neurosci 2023; 34:223-245. [PMID: 36084305 DOI: 10.1515/revneuro-2022-0065] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2022] [Accepted: 07/22/2022] [Indexed: 02/07/2023]
Abstract
Previous fMRI research identified superior temporal sulcus as central integration area for audiovisual stimuli. However, less is known about a general multisensory integration network across senses. Therefore, we conducted activation likelihood estimation meta-analysis with multiple sensory modalities to identify a common brain network. We included 49 studies covering all Aristotelian senses i.e., auditory, visual, tactile, gustatory, and olfactory stimuli. Analysis revealed significant activation in bilateral superior temporal gyrus, middle temporal gyrus, thalamus, right insula, and left inferior frontal gyrus. We assume these regions to be part of a general multisensory integration network comprising different functional roles. Here, thalamus operate as first subcortical relay projecting sensory information to higher cortical integration centers in superior temporal gyrus/sulcus while conflict-processing brain regions as insula and inferior frontal gyrus facilitate integration of incongruent information. We additionally performed meta-analytic connectivity modelling and found each brain region showed co-activations within the identified multisensory integration network. Therefore, by including multiple sensory modalities in our meta-analysis the results may provide evidence for a common brain network that supports different functional roles for multisensory integration.
Collapse
Affiliation(s)
- Sebastian Scheliga
- Department of Psychiatry, Psychotherapy and Psychosomatics, Medical Faculty RWTH Aachen University, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Thilo Kellermann
- Department of Psychiatry, Psychotherapy and Psychosomatics, Medical Faculty RWTH Aachen University, Pauwelsstraße 30, 52074 Aachen, Germany.,JARA-Institute Brain Structure Function Relationship, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Angelika Lampert
- Institute of Physiology, Medical Faculty RWTH Aachen University, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Roman Rolke
- Department of Palliative Medicine, Medical Faculty RWTH Aachen University, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Marc Spehr
- Department of Chemosensation, RWTH Aachen University, Institute for Biology, Worringerweg 3, 52074 Aachen, Germany
| | - Ute Habel
- Department of Psychiatry, Psychotherapy and Psychosomatics, Medical Faculty RWTH Aachen University, Pauwelsstraße 30, 52074 Aachen, Germany.,JARA-Institute Brain Structure Function Relationship, Pauwelsstraße 30, 52074 Aachen, Germany
| |
Collapse
|
9
|
Xu X, Li Q, Qian Y, Cai H, Zhang C, Zhao W, Zhu J, Yu Y. Genetic mechanisms underlying gray matter volume changes in patients with drug-naive first-episode schizophrenia. Cereb Cortex 2023; 33:2328-2341. [PMID: 35640648 DOI: 10.1093/cercor/bhac211] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Revised: 05/05/2022] [Accepted: 05/06/2022] [Indexed: 11/13/2022] Open
Abstract
Brain structural damage is a typical feature of schizophrenia. Investigating such disease phenotype in patients with drug-naive first-episode schizophrenia (DFSZ) may exclude the confounds of antipsychotics and illness chronicity. However, small sample sizes and marked clinical heterogeneity have precluded definitive identification of gray matter volume (GMV) changes in DFSZ as well as their underlying genetic mechanisms. Here, GMV changes in DFSZ were assessed using a neuroimaging meta-analysis of 19 original studies, including 605 patients and 637 controls. Gene expression data were derived from the Allen Human Brain Atlas and processed with a newly proposed standardized pipeline. Then, we used transcriptome-neuroimaging spatial correlations to identify genes associated with GMV changes in DFSZ, followed by a set of gene functional feature analyses. Meta-analysis revealed consistent GMV reduction in the right superior temporal gyrus, right insula and left inferior temporal gyrus in DFSZ. Moreover, we found that these GMV changes were spatially correlated with expression levels of 1,201 genes, which exhibited a wide range of functional features. Our findings may provide important insights into the genetic mechanisms underlying brain morphological abnormality in schizophrenia.
Collapse
Affiliation(s)
- Xiaotao Xu
- Department of Radiology, The Fourth Affiliated Hospital of Anhui Medical University, Hefei 230012, China.,Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China.,Research Center of Clinical Medical Imaging, Anhui Province, Hefei, 230032, China.,Anhui Provincial Institute of Translational Medicine, Hefei 230032, China
| | - Qian Li
- Department of Radiology, Chaohu Hospital of Anhui Medical University, Hefei 238000, China.,Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China.,Research Center of Clinical Medical Imaging, Anhui Province, Hefei, 230032, China.,Anhui Provincial Institute of Translational Medicine, Hefei 230032, China
| | - Yinfeng Qian
- Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China.,Research Center of Clinical Medical Imaging, Anhui Province, Hefei, 230032, China.,Anhui Provincial Institute of Translational Medicine, Hefei 230032, China
| | - Huanhuan Cai
- Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China.,Research Center of Clinical Medical Imaging, Anhui Province, Hefei, 230032, China.,Anhui Provincial Institute of Translational Medicine, Hefei 230032, China
| | - Cun Zhang
- Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China.,Research Center of Clinical Medical Imaging, Anhui Province, Hefei, 230032, China.,Anhui Provincial Institute of Translational Medicine, Hefei 230032, China
| | - Wenming Zhao
- Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China.,Research Center of Clinical Medical Imaging, Anhui Province, Hefei, 230032, China.,Anhui Provincial Institute of Translational Medicine, Hefei 230032, China
| | - Jiajia Zhu
- Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China.,Research Center of Clinical Medical Imaging, Anhui Province, Hefei, 230032, China.,Anhui Provincial Institute of Translational Medicine, Hefei 230032, China
| | - Yongqiang Yu
- Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China.,Research Center of Clinical Medical Imaging, Anhui Province, Hefei, 230032, China.,Anhui Provincial Institute of Translational Medicine, Hefei 230032, China.,Department of Radiology, Chaohu Hospital of Anhui Medical University, Hefei 238000, China.,Department of Radiology, The Fourth Affiliated Hospital of Anhui Medical University, Hefei 230012, China
| |
Collapse
|
10
|
Ma H, Cao Y, Li M, Zhan L, Xie Z, Huang L, Gao Y, Jia X. Abnormal amygdala functional connectivity and deep learning classification in multifrequency bands in autism spectrum disorder: A multisite functional magnetic resonance imaging study. Hum Brain Mapp 2023; 44:1094-1104. [PMID: 36346215 PMCID: PMC9875923 DOI: 10.1002/hbm.26141] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2022] [Revised: 10/19/2022] [Accepted: 10/25/2022] [Indexed: 11/11/2022] Open
Abstract
Previous studies have explored resting-state functional connectivity (rs-FC) of the amygdala in patients with autism spectrum disorder (ASD). However, it remains unclear whether there are frequency-specific FC alterations of the amygdala in ASD and whether FC in specific frequency bands can be used to distinguish patients with ASD from typical controls (TCs). Data from 306 patients with ASD and 314 age-matched and sex-matched TCs were collected from 28 sites in the Autism Brain Imaging Data Exchange database. The bilateral amygdala, defined as the seed regions, was used to perform seed-based FC analyses in the conventional, slow-5, and slow-4 frequency bands at each site. Image-based meta-analyses were used to obtain consistent brain regions across 28 sites in the three frequency bands. By combining generative adversarial networks and deep neural networks, a deep learning approach was applied to distinguish patients with ASD from TCs. The meta-analysis results showed frequency band specificity of FC in ASD, which was reflected in the slow-5 frequency band instead of the conventional and slow-4 frequency bands. The deep learning results showed that, compared with the conventional and slow-4 frequency bands, the slow-5 frequency band exhibited a higher accuracy of 74.73%, precision of 74.58%, recall of 75.05%, and area under the curve of 0.811 to distinguish patients with ASD from TCs. These findings may help us to understand the pathological mechanisms of ASD and provide preliminary guidance for the clinical diagnosis of ASD.
Collapse
Affiliation(s)
- Huibin Ma
- School of Information and Electronics Technology, Jiamusi University, Jiamusi, China
| | - Yikang Cao
- School of Information and Electronics Technology, Jiamusi University, Jiamusi, China
| | - Mengting Li
- College of Teacher Education, Zhejiang Normal University, Jinhua, China.,Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University, Jinhua, China
| | - Linlin Zhan
- Faculty of Western Languages, Heilongjiang University, Harbin, China
| | - Zhou Xie
- School of Information and Electronics Technology, Jiamusi University, Jiamusi, China
| | - Lina Huang
- Department of Radiology, Changshu No. 2 People's Hospital, The Affiliated Changshu Hospital of Xuzhou Medical University, Changshu, China
| | - Yanyan Gao
- College of Teacher Education, Zhejiang Normal University, Jinhua, China.,Key Laboratory of Intelligent Education Technology and Application, Zhejiang Normal University, Jinhua, China
| | - Xize Jia
- Department of Radiology, Changshu No. 2 People's Hospital, The Affiliated Changshu Hospital of Xuzhou Medical University, Changshu, China
| |
Collapse
|
11
|
Benetti S, Ferrari A, Pavani F. Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience. Front Hum Neurosci 2023; 17:1108354. [PMID: 36816496 PMCID: PMC9932987 DOI: 10.3389/fnhum.2023.1108354] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Accepted: 01/11/2023] [Indexed: 02/05/2023] Open
Abstract
In face-to-face communication, humans are faced with multiple layers of discontinuous multimodal signals, such as head, face, hand gestures, speech and non-speech sounds, which need to be interpreted as coherent and unified communicative actions. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating signals that are not connected by the communicative content. How do we achieve such an extraordinary feat, reliably, and efficiently? To address this question, we need to further move the study of human communication beyond speech-centred perspectives and promote a multimodal approach combined with interdisciplinary cooperation. Accordingly, we seek to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience into a neurocognitive model of multimodal face-to-face communication. First, we introduce a psycholinguistic framework that characterises face-to-face communication at three parallel processing levels: multiplex signals, multimodal gestalts and multilevel predictions. Second, we consider the recent proposal of a lateral neural visual pathway specifically dedicated to the dynamic aspects of social perception and reconceive it from a multimodal perspective ("lateral processing pathway"). Third, we reconcile the two frameworks into a neurocognitive model that proposes how multiplex signals, multimodal gestalts, and multilevel predictions may be implemented along the lateral processing pathway. Finally, we advocate a multimodal and multidisciplinary research approach, combining state-of-the-art imaging techniques, computational modelling and artificial intelligence for future empirical testing of our model.
Collapse
Affiliation(s)
- Stefania Benetti
- Centre for Mind/Brain Sciences, University of Trento, Trento, Italy,Interuniversity Research Centre “Cognition, Language, and Deafness”, CIRCLeS, Catania, Italy,*Correspondence: Stefania Benetti,
| | - Ambra Ferrari
- Max Planck Institute for Psycholinguistics, Donders Institute for Brain, Cognition, and Behaviour, Radboud University, Nijmegen, Netherlands
| | - Francesco Pavani
- Centre for Mind/Brain Sciences, University of Trento, Trento, Italy,Interuniversity Research Centre “Cognition, Language, and Deafness”, CIRCLeS, Catania, Italy
| |
Collapse
|
12
|
Ross LA, Molholm S, Butler JS, Bene VAD, Foxe JJ. Neural correlates of multisensory enhancement in audiovisual narrative speech perception: a fMRI investigation. Neuroimage 2022; 263:119598. [PMID: 36049699 DOI: 10.1016/j.neuroimage.2022.119598] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Revised: 08/26/2022] [Accepted: 08/28/2022] [Indexed: 11/25/2022] Open
Abstract
This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions. We expected this enhancement to emerge in regions known to underlie the integration of auditory and visual information such as the posterior superior temporal gyrus as well as parts of the broader language network, including the semantic system. To this end we presented 53 participants with a continuous narration of a story in auditory alone, visual alone, and both synchronous and asynchronous audiovisual speech conditions while recording brain activity using BOLD fMRI. We found multisensory enhancement in an extensive network of regions underlying multisensory integration and parts of the semantic network as well as extralinguistic regions not usually associated with multisensory integration, namely the primary visual cortex and the bilateral amygdala. Analysis also revealed involvement of thalamic brain regions along the visual and auditory pathways more commonly associated with early sensory processing. We conclude that under natural listening conditions, multisensory enhancement not only involves sites of multisensory integration but many regions of the wider semantic network and includes regions associated with extralinguistic sensory, perceptual and cognitive processing.
Collapse
Affiliation(s)
- Lars A Ross
- The Frederick J. and Marion A. Schindler Cognitive Neurophysiology Laboratory, The Ernest J. Del Monte Institute for Neuroscience, Department of Neuroscience, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642, USA; Department of Imaging Sciences, University of Rochester Medical Center, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642, USA; The Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA.
| | - Sophie Molholm
- The Frederick J. and Marion A. Schindler Cognitive Neurophysiology Laboratory, The Ernest J. Del Monte Institute for Neuroscience, Department of Neuroscience, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642, USA; The Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA
| | - John S Butler
- The Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA; School of Mathematical Sciences, Technological University Dublin, Kevin Street Campus, Dublin, Ireland
| | - Victor A Del Bene
- The Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA; University of Alabama at Birmingham, Heersink School of Medicine, Department of Neurology, Birmingham, Alabama, 35233, USA
| | - John J Foxe
- The Frederick J. and Marion A. Schindler Cognitive Neurophysiology Laboratory, The Ernest J. Del Monte Institute for Neuroscience, Department of Neuroscience, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642, USA; The Cognitive Neurophysiology Laboratory, Departments of Pediatrics and Neuroscience, Albert Einstein College of Medicine & Montefiore Medical Center, Bronx, New York, 10461, USA.
| |
Collapse
|
13
|
Tomasino B, Del Negro I, Garbo R, Gigli GL, D'Agostini S, Valente MR. Multisensory mental imagery of fatigue: Evidence from an fMRI study. Hum Brain Mapp 2022; 43:3143-3152. [PMID: 35315967 PMCID: PMC9189079 DOI: 10.1002/hbm.25839] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2022] [Accepted: 03/08/2022] [Indexed: 11/30/2022] Open
Abstract
Functional imaging experimental designs measuring fatigue, defined as a subjective lack of physical and/or mental energy characterizing a wide range of neurologic conditions, are still under development. Nineteen right‐handed healthy subjects (9 M and 10 F, mean age 43.15 ± 8.34 years) were evaluated by means of functional magnetic resonance imaging (fMRI), asking them to perform explicit, first‐person, mental imagery of fatigue‐related multisensory sensations. Short sentences designed to assess the principal manifestations of fatigue from the Multidimensional Fatigue Symptom Inventory were presented. Participants were asked to imagine the corresponding sensations (Sensory Imagery, SI). As a control, they had to imagine the visual scenes (Visual Imagery, VI) described in short phrases. The SI task (vs. VI task) differentially activated three areas: (i) the precuneus, which is involved in first‐person perspective taking; (ii) the left superior temporal sulcus, which is a multisensory integration area; and (iii) the left inferior frontal gyrus, known to be involved in mental imagery network. The SI fMRI task can be used to measure processing involved in mental imagery of fatigue‐related multisensory sensations.
Collapse
Affiliation(s)
- Barbara Tomasino
- Scientific Institute IRCCS "Eugenio Medea", Polo FVG, Pasian di Prato (UD), Italy
| | - Ilaria Del Negro
- Clinical Neurology, Azienda Sanitaria Universitaria Friuli Centrale, Presidio Ospedaliero Santa Maria della Misericordia, Udine, Italy
| | - Riccardo Garbo
- Clinical Neurology, Azienda Sanitaria Universitaria Friuli Centrale, Presidio Ospedaliero Santa Maria della Misericordia, Udine, Italy
| | - Gian Luigi Gigli
- Clinical Neurology, Azienda Sanitaria Universitaria Friuli Centrale, Presidio Ospedaliero Santa Maria della Misericordia, Udine, Italy.,Neurology Unit, Department of Medicine (DAME), University of Udine, Udine, Italy
| | - Serena D'Agostini
- Neuroradiology, Azienda Sanitaria Universitaria Friuli Centrale, Presidio Ospedaliero Santa Maria della Misericordia, Udine, Italy
| | - Maria Rosaria Valente
- Clinical Neurology, Azienda Sanitaria Universitaria Friuli Centrale, Presidio Ospedaliero Santa Maria della Misericordia, Udine, Italy.,Neurology Unit, Department of Medicine (DAME), University of Udine, Udine, Italy
| |
Collapse
|
14
|
Rennig J, Beauchamp MS. Intelligibility of audiovisual sentences drives multivoxel response patterns in human superior temporal cortex. Neuroimage 2022; 247:118796. [PMID: 34906712 PMCID: PMC8819942 DOI: 10.1016/j.neuroimage.2021.118796] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2021] [Revised: 11/18/2021] [Accepted: 12/08/2021] [Indexed: 11/18/2022] Open
Abstract
Regions of the human posterior superior temporal gyrus and sulcus (pSTG/S) respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech, and neural responses in pSTG/S may underlie the perceptual benefit of visual speech for the comprehension of noisy auditory speech. We examined this possibility through the lens of multivoxel pattern responses in pSTG/S. BOLD fMRI data was collected from 22 participants presented with speech consisting of English sentences presented in five different formats: visual-only; auditory with and without added auditory noise; and audiovisual with and without auditory noise. Participants reported the intelligibility of each sentence with a button press and trials were sorted post-hoc into those that were more or less intelligible. Response patterns were measured in regions of the pSTG/S identified with an independent localizer. Noisy audiovisual sentences with very similar physical properties evoked very different response patterns depending on their intelligibility. When a noisy audiovisual sentence was reported as intelligible, the pattern was nearly identical to that elicited by clear audiovisual sentences. In contrast, an unintelligible noisy audiovisual sentence evoked a pattern like that of visual-only sentences. This effect was less pronounced for noisy auditory-only sentences, which evoked similar response patterns regardless of intelligibility. The successful integration of visual and auditory speech produces a characteristic neural signature in pSTG/S, highlighting the importance of this region in generating the perceptual benefit of visual speech.
Collapse
Affiliation(s)
- Johannes Rennig
- Division of Neuropsychology, Center of Neurology, Hertie-Institute for Clinical Brain Research, University of Tübingen, Tübingen, Germany
| | - Michael S Beauchamp
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Richards Medical Research Building, A607, 3700 Hamilton Walk, Philadelphia, PA 19104-6016, United States.
| |
Collapse
|
15
|
Liu P, Sutherland M, Pollick FE. Incongruence effects in cross-modal emotional processing in autistic traits: An fMRI study. Neuropsychologia 2021; 161:107997. [PMID: 34425144 DOI: 10.1016/j.neuropsychologia.2021.107997] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2020] [Revised: 07/26/2021] [Accepted: 08/17/2021] [Indexed: 10/20/2022]
Abstract
In everyday life, emotional information is often conveyed by both the face and the voice. Consequently, information presented by one source can alter the way in which information from the other source is perceived, leading to emotional incongruence. Here, we used functional magnetic resonance imaging (fMRI) to examine neutral correlates of two different types of emotional incongruence in audiovisual processing, namely incongruence of emotion-valence and incongruence of emotion-presence. Participants were in two groups, one group with a low Autism Quotient score (LAQ) and one with a high score (HAQ). Each participant experienced emotional (happy, fearful) or neutral faces or voices while concurrently being exposed to emotional (happy, fearful) or neutral voices or faces. They were instructed to attend to either the visual or auditory track. The incongruence effect of emotion-valence was characterized by activation in a wide range of brain regions in both hemispheres involving the inferior frontal gyrus, cuneus, superior temporal gyrus, and middle frontal gyrus. The incongruence effect of emotion-presence was characterized by activation in a set of temporal and occipital regions in both hemispheres, including the middle occipital gyrus, middle temporal gyrus and inferior temporal gyrus. In addition, the present study identified greater recruitment of the right inferior parietal lobule in perceiving audio-visual emotional expressions in HAQ individuals, as compared to the LAQ individuals. Depending on face or voice-to-be attended, different patterns of emotional incongruence were found between the two groups. Specifically, the HAQ group tend to show more incidental processing to visual information whilst the LAQ group tend to show more incidental processing to auditory information during the crossmodal emotional incongruence decoding. These differences might be attributed to different attentional demands and different processing strategies between the two groups.
Collapse
Affiliation(s)
- Peipei Liu
- Department of Psychology, Sun Yat-Sen University, Guangzhou, 510006, China; School of Psychology, University of Glasgow, Glasgow, G12 8QB, UK; School of Education, University of Glasgow, Glasgow, G3 6NH, UK
| | | | - Frank E Pollick
- School of Psychology, University of Glasgow, Glasgow, G12 8QB, UK.
| |
Collapse
|
16
|
Sievers B, Parkinson C, Kohler PJ, Hughes JM, Fogelson SV, Wheatley T. Visual and auditory brain areas share a representational structure that supports emotion perception. Curr Biol 2021; 31:5192-5203.e4. [PMID: 34644547 DOI: 10.1016/j.cub.2021.09.043] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Revised: 07/07/2021] [Accepted: 09/16/2021] [Indexed: 11/18/2022]
Abstract
Emotionally expressive music and dance occur together across the world. This may be because features shared across the senses are represented the same way even in different sensory brain areas, putting music and movement in directly comparable terms. These shared representations may arise from a general need to identify environmentally relevant combinations of sensory features, particularly those that communicate emotion. To test the hypothesis that visual and auditory brain areas share a representational structure, we created music and animation stimuli with crossmodally matched features expressing a range of emotions. Participants confirmed that each emotion corresponded to a set of features shared across music and movement. A subset of participants viewed both music and animation during brain scanning, revealing that representations in auditory and visual brain areas were similar to one another. This shared representation captured not only simple stimulus features but also combinations of features associated with emotion judgments. The posterior superior temporal cortex represented both music and movement using this same structure, suggesting supramodal abstraction of sensory content. Further exploratory analysis revealed that early visual cortex used this shared representational structure even when stimuli were presented auditorily. We propose that crossmodally shared representations support mutually reinforcing dynamics across auditory and visual brain areas, facilitating crossmodal comparison. These shared representations may help explain why emotions are so readily perceived and why some dynamic emotional expressions can generalize across cultural contexts.
Collapse
Affiliation(s)
- Beau Sievers
- Department of Psychology, Harvard University, Cambridge, MA 02138, USA; Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH 03755, USA.
| | - Carolyn Parkinson
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095, USA; Brain Research Institute, University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Peter J Kohler
- Department of Psychology, York University, Toronto, ON, Canada; Centre for Vision Research, York University, Toronto, ON, Canada
| | | | | | - Thalia Wheatley
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH 03755, USA; Santa Fe Institute, Santa Fe, NM 87501, USA.
| |
Collapse
|
17
|
Ito T, Ohashi H, Gracco VL. Somatosensory contribution to audio-visual speech processing. Cortex 2021; 143:195-204. [PMID: 34450567 DOI: 10.1016/j.cortex.2021.07.013] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2021] [Revised: 07/20/2021] [Accepted: 07/28/2021] [Indexed: 10/20/2022]
Abstract
Recent studies have demonstrated that the auditory speech perception of a listener can be modulated by somatosensory input applied to the facial skin suggesting that perception is an embodied process. However, speech perception is a multisensory process involving both the auditory and visual modalities. It is unknown whether and to what extent somatosensory stimulation to the facial skin modulates audio-visual speech perception. If speech perception is an embodied process, then somatosensory stimulation applied to the perceiver should influence audio-visual speech processing. Using the McGurk effect (the perceptual illusion that occurs when a sound is paired with the visual representation of a different sound, resulting in the perception of a third sound) we tested the prediction using a simple behavioral paradigm and at the neural level using event-related potentials (ERPs) and their cortical sources. We recorded ERPs from 64 scalp sites in response to congruent and incongruent audio-visual speech randomly presented with and without somatosensory stimulation associated with facial skin deformation. Subjects judged whether the production was /ba/ or not under all stimulus conditions. In the congruent audio-visual condition subjects identifying the sound as /ba/, but not in the incongruent condition consistent with the McGurk effect. Concurrent somatosensory stimulation improved the ability of participants to more correctly identify the production as /ba/ relative to the non-somatosensory condition in both congruent and incongruent conditions. ERP in response to the somatosensory stimulation for the incongruent condition reliably diverged 220 msec after stimulation onset. Cortical sources were estimated around the left anterior temporal gyrus, the right middle temporal gyrus, the right posterior superior temporal lobe and the right occipital region. The results demonstrate a clear multisensory convergence of somatosensory and audio-visual processing in both behavioral and neural processing consistent with the perspective that speech perception is a self-referenced, sensorimotor process.
Collapse
Affiliation(s)
- Takayuki Ito
- University Grenoble-Alpes, CNRS, Grenoble-INP, GIPSA-Lab, Saint Martin D'heres Cedex, France; Haskins Laboratories, New Haven, CT, USA.
| | | | - Vincent L Gracco
- Haskins Laboratories, New Haven, CT, USA; McGill University, Montréal, QC, Canada
| |
Collapse
|
18
|
Abstract
Coordination between different sensory systems is a necessary element of sensory processing. Where and how signals from different sense organs converge onto common neural circuitry have become topics of increasing interest in recent years. In this article, we focus specifically on visual-auditory interactions in areas of the mammalian brain that are commonly considered to be auditory in function. The auditory cortex and inferior colliculus are two key points of entry where visual signals reach the auditory pathway, and both contain visual- and/or eye movement-related signals in humans and other animals. The visual signals observed in these auditory structures reflect a mixture of visual modulation of auditory-evoked activity and visually driven responses that are selective for stimulus location or features. These key response attributes also appear in the classic visual pathway but may play a different role in the auditory pathway: to modify auditory rather than visual perception. Finally, while this review focuses on two particular areas of the auditory pathway where this question has been studied, robust descending as well as ascending connections within this pathway suggest that undiscovered visual signals may be present at other stages as well. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Collapse
Affiliation(s)
- Meredith N Schmehl
- Department of Neurobiology, Duke University, Durham, North Carolina 27708, USA; , .,Center for Cognitive Neuroscience, Duke University, Durham, North Carolina 27708, USA.,Duke Institute for Brain Sciences, Duke University, Durham, North Carolina 27708, USA
| | - Jennifer M Groh
- Department of Neurobiology, Duke University, Durham, North Carolina 27708, USA; , .,Department of Psychology & Neuroscience, Duke University, Durham, North Carolina 27708, USA.,Department of Computer Science, Duke University, Durham, North Carolina 27708, USA.,Department of Biomedical Engineering, Duke University, Durham, North Carolina 27708, USA.,Center for Cognitive Neuroscience, Duke University, Durham, North Carolina 27708, USA.,Duke Institute for Brain Sciences, Duke University, Durham, North Carolina 27708, USA
| |
Collapse
|
19
|
Beauchamp MS. Face and voice perception: Monkey see, monkey hear. Curr Biol 2021; 31:R435-R437. [PMID: 33974868 DOI: 10.1016/j.cub.2021.02.060] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
Primate brains contain specialized areas for perceiving social cues. New research shows that only some of these areas integrate visual faces with auditory voices.
Collapse
Affiliation(s)
- Michael S Beauchamp
- Department of Neurosurgery, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA 19104, USA.
| |
Collapse
|
20
|
Processing communicative facial and vocal cues in the superior temporal sulcus. Neuroimage 2020; 221:117191. [PMID: 32711066 DOI: 10.1016/j.neuroimage.2020.117191] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2020] [Revised: 07/14/2020] [Accepted: 07/19/2020] [Indexed: 11/20/2022] Open
Abstract
Facial and vocal cues provide critical social information about other humans, including their emotional and attentional states and the content of their speech. Recent work has shown that the face-responsive region of posterior superior temporal sulcus ("fSTS") also responds strongly to vocal sounds. Here, we investigate the functional role of this region and the broader STS by measuring responses to a range of face movements, vocal sounds, and hand movements using fMRI. We find that the fSTS responds broadly to different types of audio and visual face action, including both richly social communicative actions, as well as minimally social noncommunicative actions, ruling out hypotheses of specialization for processing speech signals, or communicative signals more generally. Strikingly, however, responses to hand movements were very low, whether communicative or not, indicating a specific role in the analysis of face actions (facial and vocal), not a general role in the perception of any human action. Furthermore, spatial patterns of response in this region were able to decode communicative from noncommunicative face actions, both within and across modality (facial/vocal cues), indicating sensitivity to an abstract social dimension. These functional properties of the fSTS contrast with a region of middle STS that has a selective, largely unimodal auditory response to speech sounds over both communicative and noncommunicative vocal nonspeech sounds, and nonvocal sounds. Region of interest analyses were corroborated by a data-driven independent component analysis, identifying face-voice and auditory speech responses as dominant sources of voxelwise variance across the STS. These results suggest that the STS contains separate processing streams for the audiovisual analysis of face actions and auditory speech processing.
Collapse
|
21
|
Prosodic influence in face emotion perception: evidence from functional near-infrared spectroscopy. Sci Rep 2020; 10:14345. [PMID: 32873844 PMCID: PMC7462865 DOI: 10.1038/s41598-020-71266-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2018] [Accepted: 07/30/2020] [Indexed: 11/08/2022] Open
Abstract
Emotion is communicated via the integration of concurrently presented information from multiple information channels, such as voice, face, gesture and touch. This study investigated the neural and perceptual correlates of emotion perception as influenced by facial and vocal information by measuring changes in oxygenated hemoglobin (HbO) using functional near-infrared spectroscopy (fNIRS) and acquiring psychometrics. HbO activity was recorded from 103 channels while participants ([Formula: see text], [Formula: see text]) were presented with vocalizations produced in either a happy, angry or neutral prosody. Voices were presented alone or paired with an emotional face and compared with a face-only condition. Behavioral results indicated that when voices were paired with faces, a bias in the direction of the emotion of the voice was present. Subjects' responses also showed greater variance and longer reaction times when responding to the bimodal conditions when compared to the face-only condition. While both the happy and angry prosody conditions exhibited right lateralized increases in HbO compared to the neutral condition, these activations were segregated into posterior-anterior subdivisions by emotion. Specific emotional prosodies may therefore differentially influence emotion perception, with happy voices exhibiting posterior activity in receptive emotion areas and angry voices displaying activity in anterior expressive emotion areas.
Collapse
|
22
|
Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution. J Neurosci 2020; 40:6938-6948. [PMID: 32727820 PMCID: PMC7470920 DOI: 10.1523/jneurosci.0279-20.2020] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2020] [Revised: 06/01/2020] [Accepted: 06/02/2020] [Indexed: 12/22/2022] Open
Abstract
Experimentalists studying multisensory integration compare neural responses to multisensory stimuli with responses to the component modalities presented in isolation. This procedure is problematic for multisensory speech perception since audiovisual speech and auditory-only speech are easily intelligible but visual-only speech is not. To overcome this confound, we developed intracranial encephalography (iEEG) deconvolution. Individual stimuli always contained both auditory and visual speech, but jittering the onset asynchrony between modalities allowed for the time course of the unisensory responses and the interaction between them to be independently estimated. We applied this procedure to electrodes implanted in human epilepsy patients (both male and female) over the posterior superior temporal gyrus (pSTG), a brain area known to be important for speech perception. iEEG deconvolution revealed sustained positive responses to visual-only speech and larger, phasic responses to auditory-only speech. Confirming results from scalp EEG, responses to audiovisual speech were weaker than responses to auditory-only speech, demonstrating a subadditive multisensory neural computation. Leveraging the spatial resolution of iEEG, we extended these results to show that subadditivity is most pronounced in more posterior aspects of the pSTG. Across electrodes, subadditivity correlated with visual responsiveness, supporting a model in which visual speech enhances the efficiency of auditory speech processing in pSTG. The ability to separate neural processes may make iEEG deconvolution useful for studying a variety of complex cognitive and perceptual tasks.SIGNIFICANCE STATEMENT Understanding speech is one of the most important human abilities. Speech perception uses information from both the auditory and visual modalities. It has been difficult to study neural responses to visual speech because visual-only speech is difficult or impossible to comprehend, unlike auditory-only and audiovisual speech. We used intracranial encephalography deconvolution to overcome this obstacle. We found that visual speech evokes a positive response in the human posterior superior temporal gyrus, enhancing the efficiency of auditory speech processing.
Collapse
|
23
|
Arsalidou M, Yaple Z, Jurcik T, Ushakov V. Cognitive Brain Signatures of Youth With Early Onset and Relatives With Schizophrenia: Evidence From fMRI Meta-analyses. Schizophr Bull 2020; 46:857-868. [PMID: 31978222 PMCID: PMC7345811 DOI: 10.1093/schbul/sbz130] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/23/2023]
Abstract
Deficits in cognitive function are a major characteristic of schizophrenia. Many functional magnetic resonance imaging (fMRI) studies examine brain correlates of cognitive function in adults with schizophrenia, showing altered implication of associative areas such as the prefrontal cortex and temporal cortex. fMRI studies also examine brain representation of cognitive function in adolescents with early onset schizophrenia and those at risk of the disorder, yet results are often inconsistent. We compile and analyze data from eligible fMRI studies using quantitative meta-analyses to reveal concordant brain activity associated with adolescent relatives of patients with schizophrenia and those with early onset schizophrenia. Results show similar functional hubs of brain activity (eg, precuneus) yet in opposite hemispheres and clusters in ventrolateral rather than dorsolateral prefrontal cortices. Other areas of altered implication include the middle temporal gyrus, insula, and cerebellum. We discuss the findings in reference to the protracted maturation of the prefrontal cortex and possible effects due to the medication status of the two groups.
Collapse
Affiliation(s)
- Marie Arsalidou
- Department of Psychology, National Research University Higher School of Economics, Moscow, Russian Federation,Department of Psychology, Faculty of Health, York University, Toronto, ON, Canada,To whom correspondence should be addressed; Armyanskiy per. 4, c2, Moscow, 101000, room 406; tel: 1786-505-9779, e-mail: ; ;
| | - Zachary Yaple
- Department of Psychology, National University of Singapore, Singapore
| | - Tomas Jurcik
- Department of Psychology, National Research University Higher School of Economics, Moscow, Russian Federation
| | - Vadim Ushakov
- Kurchatov Department of NBICS-nature-like technologies, National Research Centre Kurchatov Institute, Moscow, Russian Federation,Department of Cybernetics, National Research Nuclear University “MEPhI”, Moscow, Russian Federation
| |
Collapse
|
24
|
Young AW, Frühholz S, Schweinberger SR. Face and Voice Perception: Understanding Commonalities and Differences. Trends Cogn Sci 2020; 24:398-410. [DOI: 10.1016/j.tics.2020.02.001] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/08/2019] [Revised: 01/16/2020] [Accepted: 02/03/2020] [Indexed: 01/01/2023]
|
25
|
Borowiak K, Maguinness C, von Kriegstein K. Dorsal-movement and ventral-form regions are functionally connected during visual-speech recognition. Hum Brain Mapp 2020; 41:952-972. [PMID: 31749219 PMCID: PMC7267922 DOI: 10.1002/hbm.24852] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2019] [Revised: 09/03/2019] [Accepted: 10/21/2019] [Indexed: 01/17/2023] Open
Abstract
Faces convey social information such as emotion and speech. Facial emotion processing is supported via interactions between dorsal-movement and ventral-form visual cortex regions. Here, we explored, for the first time, whether similar dorsal-ventral interactions (assessed via functional connectivity), might also exist for visual-speech processing. We then examined whether altered dorsal-ventral connectivity is observed in adults with high-functioning autism spectrum disorder (ASD), a disorder associated with impaired visual-speech recognition. We acquired functional magnetic resonance imaging (fMRI) data with concurrent eye tracking in pairwise matched control and ASD participants. In both groups, dorsal-movement regions in the visual motion area 5 (V5/MT) and the temporal visual speech area (TVSA) were functionally connected to ventral-form regions (i.e., the occipital face area [OFA] and the fusiform face area [FFA]) during the recognition of visual speech, in contrast to the recognition of face identity. Notably, parts of this functional connectivity were decreased in the ASD group compared to the controls (i.e., right V5/MT-right OFA, left TVSA-left FFA). The results confirmed our hypothesis that functional connectivity between dorsal-movement and ventral-form regions exists during visual-speech processing. Its partial dysfunction in ASD might contribute to difficulties in the recognition of dynamic face information relevant for successful face-to-face communication.
Collapse
Affiliation(s)
- Kamila Borowiak
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität DresdenDresdenGermany
- Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany
- Berlin School of Mind and Brain, Humboldt University of BerlinBerlinGermany
| | - Corrina Maguinness
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität DresdenDresdenGermany
- Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany
| | - Katharina von Kriegstein
- Chair of Cognitive and Clinical Neuroscience, Faculty of Psychology, Technische Universität DresdenDresdenGermany
- Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany
| |
Collapse
|
26
|
Proverbio AM, Camporeale E, Brusa A. Multimodal Recognition of Emotions in Music and Facial Expressions. Front Hum Neurosci 2020; 14:32. [PMID: 32116613 PMCID: PMC7027335 DOI: 10.3389/fnhum.2020.00032] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2019] [Accepted: 01/23/2020] [Indexed: 01/24/2023] Open
Abstract
The aim of the study was to investigate the neural processing of congruent vs. incongruent affective audiovisual information (facial expressions and music) by means of ERPs (Event Related Potentials) recordings. Stimuli were 200 infant faces displaying Happiness, Relaxation, Sadness, Distress and 32 piano musical pieces conveying the same emotional states (as specifically assessed). Music and faces were presented simultaneously, and paired so that in half cases they were emotionally congruent or incongruent. Twenty subjects were told to pay attention and respond to infrequent targets (adult neutral faces) while their EEG was recorded from 128 channels. The face-related N170 (160-180 ms) component was the earliest response affected by the emotional content of faces (particularly by distress), while visual P300 (250-450 ms) and auditory N400 (350-550 ms) responses were specifically modulated by the emotional content of both facial expressions and musical pieces. Face/music emotional incongruence elicited a wide N400 negativity indicating the detection of a mismatch in the expressed emotion. A swLORETA inverse solution applied to N400 (difference wave Incong. - Cong.), showed the crucial role of Inferior and Superior Temporal Gyri in the multimodal representation of emotional information extracted from faces and music. Furthermore, the prefrontal cortex (superior and medial, BA 10) was also strongly active, possibly supporting working memory. The data hints at a common system for representing emotional information derived by social cognition and music processing, including uncus and cuneus.
Collapse
|
27
|
REN Y, XU Z, WANG T, YANG W. AGE-RELATED ALTERATIONS IN AUDIOVISUAL INTEGRATION: A BRIEF OVERVIEW. PSYCHOLOGIA 2020. [DOI: 10.2117/psysoc.2020-a002] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Affiliation(s)
- Yanna REN
- Guizhou University of Chinese Medicine
| | - Zhihan XU
- Okayama University
- Ningbo University of Technology
| | - Tao WANG
- Guizhou Light Industry Technical College
| | | |
Collapse
|
28
|
Gao C, Weber CE, Shinkareva SV. The brain basis of audiovisual affective processing: Evidence from a coordinate-based activation likelihood estimation meta-analysis. Cortex 2019; 120:66-77. [DOI: 10.1016/j.cortex.2019.05.016] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2019] [Revised: 05/03/2019] [Accepted: 05/28/2019] [Indexed: 01/19/2023]
|
29
|
Faces and voices in the brain: A modality-general person-identity representation in superior temporal sulcus. Neuroimage 2019; 201:116004. [DOI: 10.1016/j.neuroimage.2019.07.017] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2018] [Revised: 05/17/2019] [Accepted: 07/07/2019] [Indexed: 11/18/2022] Open
|
30
|
Tietze FA, Hundertmark L, Roy M, Zerr M, Sinke C, Wiswede D, Walter M, Münte TF, Szycik GR. Auditory Deficits in Audiovisual Speech Perception in Adult Asperger's Syndrome: fMRI Study. Front Psychol 2019; 10:2286. [PMID: 31649597 PMCID: PMC6795762 DOI: 10.3389/fpsyg.2019.02286] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2019] [Accepted: 09/24/2019] [Indexed: 01/23/2023] Open
Abstract
Audiovisual (AV) integration deficits have been proposed to underlie difficulties in speech perception in Asperger’s syndrome (AS). It is not known, if the AV deficits are related to alterations in sensory processing at the level of unisensory processing or at levels of conjoint multisensory processing. Functional Magnetic-resonance images (MRI) was performed in 16 adult subjects with AS and 16 healthy controls (HC) matched for age, gender, and verbal IQ as they were exposed to disyllabic AV congruent and AV incongruent nouns. A simple semantic categorization task was used to ensure subjects’ attention to the stimuli. The left auditory cortex (BA41) showed stronger activation in HC than in subjects with AS with no interaction regarding AV congruency. This suggests that alterations in auditory processing in unimodal low-level areas underlie AV speech perception deficits in AS. Whether this is signaling a difficulty in the deployment of attention remains to be demonstrated.
Collapse
Affiliation(s)
- Fabian-Alexander Tietze
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany
| | - Laura Hundertmark
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany
| | - Mandy Roy
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany.,Asklepios Clinic North - Ochsenzoll, Hamburg, Germany
| | - Michael Zerr
- Department of Psychosomatic Medicine and Psychotherapy, Hannover Medical School, Hanover, Germany
| | - Christopher Sinke
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany
| | - Daniel Wiswede
- Institute of Psychology II, University of Lübeck, Lübeck, Germany.,Department of Neurology, University of Lübeck, Lübeck, Germany
| | - Martin Walter
- Department of Psychiatry and Psychotherapy, University of Tübingen, Tübingen, Germany
| | - Thomas F Münte
- Institute of Psychology II, University of Lübeck, Lübeck, Germany.,Department of Neurology, University of Lübeck, Lübeck, Germany
| | - Gregor R Szycik
- Department of Psychiatry, Social Psychiatry and Psychotherapy, Hannover Medical School, Hanover, Germany
| |
Collapse
|
31
|
Neural basis of romantic partners' decisions about participation in leisure activity. Sci Rep 2019; 9:14448. [PMID: 31595015 PMCID: PMC6783572 DOI: 10.1038/s41598-019-51038-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2018] [Accepted: 09/23/2019] [Indexed: 11/12/2022] Open
Abstract
Leisure activity is one of key ingredients for individual happiness and life satisfaction. Enjoying leisure activity with one’s partner can increase marital satisfaction. This study aimed to identify the neural basis of making decisions on participation in a leisure activity with one’s romantic partner as well as the relationship between leisure activity and satisfaction with life. Thirty-seven soon-to-be married heterosexual couples were participated in functional MRI while deciding participation in specific leisure activities in the individual, partner, with-friend, and with-partner conditions. We constructed analysis of variance models and investigated couple characteristics such as personality similarity, leisure activity matching rate, and spatial similarity in the bilateral frontoparietal network. The results showed decreased activity in the bilateral hippocampus during the task in the with-partner condition. Individual leisure activity was correlated with quality of life in males, whereas participation in leisure activity might require more cognitive loading on the dorsolateral prefrontal cortex in females. The leisure activity matching rate was correlated with courtship period, personality similarity, and spatial similarity of the right frontoparietal network during the task. These findings suggest that although there are different activation pattern in making decisions on leisure activity between romantic couples, spatial similarity of the partner’s social brain networks may be a marker that predicts how well the couple enjoys leisure activity together. In addition, our couples’ data analysis provides a scientific basis for the saying that romantic couples become more similar the longer they are together.
Collapse
|
32
|
O'Sullivan AE, Lim CY, Lalor EC. Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations. Eur J Neurosci 2019; 50:3282-3295. [PMID: 31013361 DOI: 10.1111/ejn.14425] [Citation(s) in RCA: 23] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2018] [Revised: 03/25/2019] [Accepted: 04/17/2019] [Indexed: 11/30/2022]
Abstract
Recent work using electroencephalography has applied stimulus reconstruction techniques to identify the attended speaker in a cocktail party environment. The success of these approaches has been primarily based on the ability to detect cortical tracking of the acoustic envelope at the scalp level. However, most studies have ignored the effects of visual input, which is almost always present in naturalistic scenarios. In this study, we investigated the effects of visual input on envelope-based cocktail party decoding in two multisensory cocktail party situations: (a) Congruent AV-facing the attended speaker while ignoring another speaker represented by the audio-only stream and (b) Incongruent AV (eavesdropping)-attending the audio-only speaker while looking at the unattended speaker. We trained and tested decoders for each condition separately and found that we can successfully decode attention to congruent audiovisual speech and can also decode attention when listeners were eavesdropping, i.e., looking at the face of the unattended talker. In addition to this, we found alpha power to be a reliable measure of attention to the visual speech. Using parieto-occipital alpha power, we found that we can distinguish whether subjects are attending or ignoring the speaker's face. Considering the practical applications of these methods, we demonstrate that with only six near-ear electrodes we can successfully determine the attended speech. This work extends the current framework for decoding attention to speech to more naturalistic scenarios, and in doing so provides additional neural measures which may be incorporated to improve decoding accuracy.
Collapse
Affiliation(s)
- Aisling E O'Sullivan
- School of Engineering, Trinity Centre for Bioengineering and Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin 2, Ireland
| | - Chantelle Y Lim
- Department of Biomedical Engineering, University of Rochester, Rochester, New York
| | - Edmund C Lalor
- School of Engineering, Trinity Centre for Bioengineering and Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin 2, Ireland.,Department of Biomedical Engineering, University of Rochester, Rochester, New York.,Department of Neuroscience, Del Monte Institute for Neuroscience, University of Rochester, Rochester, New York
| |
Collapse
|
33
|
A functional MRI investigation of crossmodal interference in an audiovisual Stroop task. PLoS One 2019; 14:e0210736. [PMID: 30645634 PMCID: PMC6333399 DOI: 10.1371/journal.pone.0210736] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2018] [Accepted: 01/01/2019] [Indexed: 01/08/2023] Open
Abstract
The visual color-word Stroop task is widely used in clinical and research settings as a measure of cognitive control. Numerous neuroimaging studies have used color-word Stroop tasks to investigate the neural resources supporting cognitive control, but to our knowledge all have used unimodal (typically visual) Stroop paradigms. Thus, it is possible that this classic measure of cognitive control is not capturing the resources involved in multisensory cognitive control. The audiovisual integration and crossmodal correspondence literatures identify regions sensitive to congruency of auditory and visual stimuli, but it is unclear how these regions relate to the unimodal cognitive control literature. In this study we aimed to identify brain regions engaged by crossmodal cognitive control during an audiovisual color-word Stroop task, and how they relate to previous unimodal Stroop and audiovisual integration findings. First, we replicated previous behavioral audiovisual Stroop findings in an fMRI-adapted audiovisual Stroop paradigm: incongruent visual information increased reaction time towards an auditory stimulus and congruent visual information decreased reaction time. Second, we investigated the brain regions supporting cognitive control during an audiovisual color-word Stroop task using fMRI. Similar to unimodal cognitive control tasks, a left superior parietal region exhibited an interference effect of visual information on the auditory stimulus. This superior parietal region was also identified using a standard audiovisual integration localizing procedure, indicating that audiovisual integration resources are sensitive to cognitive control demands. Facilitation of the auditory stimulus by congruent visual information was found in posterior superior temporal cortex, including in the posterior STS which has been found to support audiovisual integration. The dorsal anterior cingulate cortex, often implicated in unimodal Stroop tasks, was not modulated by the audiovisual Stroop task. Overall the findings indicate that an audiovisual color-word Stroop task engages overlapping resources with audiovisual integration and overlapping but distinct resources compared to unimodal Stroop tasks.
Collapse
|
34
|
Kim JY, Jeon H, Kwon A, Jin MJ, Lee SH, Chung YC. Self-Awareness of Psychopathology and Brain Volume in Patients With First Episode Psychosis. Front Psychiatry 2019; 10:839. [PMID: 31803084 PMCID: PMC6873658 DOI: 10.3389/fpsyt.2019.00839] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/25/2019] [Accepted: 10/22/2019] [Indexed: 01/06/2023] Open
Abstract
Memory impairment, excessive rumination, and increased interpersonal sensitivity are major characteristics of high psychosis risk or first episode psychosis (FEP). Herein, we investigated the relationship between brain volume and self-awareness of psychopathology in patients with FEP. All participants (FEP: 34 and HCs: 34) completed clinical assessments and the following self-reported psychopathology evaluations: prospective and retrospective memory questionnaire (PRMQ), ruminative response scale (RRS), and interpersonal sensitivity measure (IPSM). Structural magnetic resonance imaging was then conducted. The PRMQ, RRS, and IPSM scores were significantly higher in the FEP group than in the healthy controls (HCs). The volumes of the amygdala, hippocampus, and superior temporal gyrus (STG) were significantly lower in the FEP group than in the HCs. There was a significant group-dependent moderation effect between self-awareness of psychopathology (PRMQ, RRS, and IPSM scores) and right STG (rSTG) volume. In the FEP group, self-awareness of psychopathology was positively associated with rSTG volume, while in the HCs, this correlation was negative. Our results indicate that self-awareness of psychopathology impacts rSTG volume in the opposite direction between patients with FEP and HCs. In patients with FEP, awareness of impairment may induce increases in rSTG brain volume. However, HCs showed decreased rSTG volume when they were aware of impairment.
Collapse
Affiliation(s)
- Jeong-Youn Kim
- Clinical Emotion and Cognition Research Laboratory, Inje University, Goyang, South Korea
| | - Hyeonjin Jeon
- Clinical Emotion and Cognition Research Laboratory, Inje University, Goyang, South Korea
| | - Aeran Kwon
- Clinical Emotion and Cognition Research Laboratory, Inje University, Goyang, South Korea
| | - Min Jin Jin
- Clinical Emotion and Cognition Research Laboratory, Inje University, Goyang, South Korea.,Department of Psychology, Chung-Ang University, Seoul, South Korea
| | - Seung-Hwan Lee
- Clinical Emotion and Cognition Research Laboratory, Inje University, Goyang, South Korea.,Department of Psychiatry, Inje University, Ilsan-Paik Hospital, Goyang, South Korea
| | - Young-Chul Chung
- Department of Psychiatry, Chonbuk National University Medical School, Jeonju, South Korea
| |
Collapse
|
35
|
Rennig J, Beauchamp MS. Free viewing of talking faces reveals mouth and eye preferring regions of the human superior temporal sulcus. Neuroimage 2018; 183:25-36. [PMID: 30092347 PMCID: PMC6214361 DOI: 10.1016/j.neuroimage.2018.08.008] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2018] [Revised: 07/31/2018] [Accepted: 08/05/2018] [Indexed: 01/22/2023] Open
Abstract
During face-to-face communication, the mouth of the talker is informative about speech content, while the eyes of the talker convey other information, such as gaze location. Viewers most often fixate either the mouth or the eyes of the talker's face, presumably allowing them to sample these different sources of information. To study the neural correlates of this process, healthy humans freely viewed talking faces while brain activity was measured with BOLD fMRI and eye movements were recorded with a video-based eye tracker. Post hoc trial sorting was used to divide the data into trials in which participants fixated the mouth of the talker and trials in which they fixated the eyes. Although the audiovisual stimulus was identical, the two trials types evoked differing responses in subregions of the posterior superior temporal sulcus (pSTS). The anterior pSTS preferred trials in which participants fixated the mouth of the talker while the posterior pSTS preferred fixations on the eye of the talker. A second fMRI experiment demonstrated that anterior pSTS responded more strongly to auditory and audiovisual speech than posterior pSTS eye-preferring regions. These results provide evidence for functional specialization within the pSTS under more realistic viewing and stimulus conditions than in previous neuroimaging studies.
Collapse
Affiliation(s)
- Johannes Rennig
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, TX, USA
| | - Michael S Beauchamp
- Department of Neurosurgery and Core for Advanced MRI, Baylor College of Medicine, Houston, TX, USA.
| |
Collapse
|
36
|
Schobert AK, Corradi-Dell’Acqua C, Frühholz S, van der Zwaag W, Vuilleumier P. Functional organization of face processing in the human superior temporal sulcus: a 7T high-resolution fMRI study. Soc Cogn Affect Neurosci 2018; 13:102-113. [PMID: 29140527 PMCID: PMC5793830 DOI: 10.1093/scan/nsx119] [Citation(s) in RCA: 26] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2016] [Accepted: 10/16/2017] [Indexed: 11/14/2022] Open
Abstract
The superior temporal sulcus (STS) is a major component of the human face perception network, implicated in processing dynamic changeable aspects of faces. However, it remains unknown whether STS holds functionally segregated subdivisions for different categories of facial movements. We used high-resolution functional magnetic resonance imaging (fMRI) at 7T in 16 volunteers to compare STS activation with faces displaying angry or happy expressions, eye-gaze shifts and lip-speech movements. Combining univariate and multivariate analyses, we show a systematic topological organization within STS, with gaze-related activity predominating in the most posterior and superior sector, speech-related activity in the anterior sector and emotional expressions represented in the intermediate middle STS. Right STS appeared to hold a finer functional segregation between all four types of facial movements, and best discriminative abilities within the face-selective posterior STS (pSTS). Conversely, left STS showed greater overlap between conditions, with a lack of distinction between mouth movements associated to speech or happy expression and better discriminative abilities (for gaze and speech vs emotion conditions) outside pSTS. Differential sensitivity to upper (eye) or lower (mouth) facial features may contribute to, but does not appear to fully account for, these response patterns.
Collapse
Affiliation(s)
- Anne-Kathrin Schobert
- Department of Neurosciences and Clinic of Neurology, University Medical Center, CH-1211 Geneva, Switzerland
- Swiss Centre for Affective Sciences
| | - Corrado Corradi-Dell’Acqua
- Department of Neurosciences and Clinic of Neurology, University Medical Center, CH-1211 Geneva, Switzerland
- Swiss Centre for Affective Sciences
- Department of Psychology, FPSE, University of Geneva, CH-1211 Geneva, Switzerland
| | - Sascha Frühholz
- Swiss Centre for Affective Sciences
- Department of Psychology, University of Zurich, Zurich, Switzerland
- Neuroscience Center Zurich, Zurich, Switzerland
- Zurich Center for Integrative Human Physiology (ZIHP), Zurich, Switzerland
| | - Wietske van der Zwaag
- CIBM, EPFL, Lausanne, VD, Switzerland
- Spinoza Centre for Neuroimaging, Amsterdam, The Netherlands
| | - Patrik Vuilleumier
- Department of Neurosciences and Clinic of Neurology, University Medical Center, CH-1211 Geneva, Switzerland
- Swiss Centre for Affective Sciences
- Correspondence should be addressed to Patrik Vuilleumier, Laboratory for Neurology and Imaging of Cognition, Department of Neurosciences, 1 rue Michel-Servet, 1211 Geneva, Switzerland. E-mail:
| |
Collapse
|
37
|
Devesse A, Dudek A, van Wieringen A, Wouters J. Speech intelligibility of virtual humans. Int J Audiol 2018; 57:908-916. [DOI: 10.1080/14992027.2018.1511922] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/28/2022]
Affiliation(s)
- Annelies Devesse
- KU Leuven, – University of Leuven, Department of Neurosciences, ExpORL, Leuven, Belgium
| | - Alexander Dudek
- KU Leuven, – University of Leuven, Department of Neurosciences, ExpORL, Leuven, Belgium
| | - Astrid van Wieringen
- KU Leuven, – University of Leuven, Department of Neurosciences, ExpORL, Leuven, Belgium
| | - Jan Wouters
- KU Leuven, – University of Leuven, Department of Neurosciences, ExpORL, Leuven, Belgium
| |
Collapse
|
38
|
Adult dyslexic readers benefit less from visual input during audiovisual speech processing: fMRI evidence. Neuropsychologia 2018; 117:454-471. [DOI: 10.1016/j.neuropsychologia.2018.07.009] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2017] [Revised: 06/13/2018] [Accepted: 07/06/2018] [Indexed: 11/19/2022]
|
39
|
Jesse A, Bartoli M. Learning to recognize unfamiliar talkers: Listeners rapidly form representations of facial dynamic signatures. Cognition 2018; 176:195-208. [DOI: 10.1016/j.cognition.2018.03.018] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2017] [Revised: 03/13/2018] [Accepted: 03/21/2018] [Indexed: 11/25/2022]
|
40
|
Proverbio AM, Raso G, Zani A. Electrophysiological Indexes of Incongruent Audiovisual Phonemic Processing: Unraveling the McGurk Effect. Neuroscience 2018; 385:215-226. [PMID: 29932985 DOI: 10.1016/j.neuroscience.2018.06.021] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2017] [Revised: 06/11/2018] [Accepted: 06/12/2018] [Indexed: 11/15/2022]
Abstract
In this study the timing of electromagnetic signals recorded during incongruent and congruent audiovisual (AV) stimulation in 14 Italian healthy volunteers was examined. In a previous study (Proverbio et al., 2016) we investigated the McGurk effect in the Italian language and found out which visual and auditory inputs provided the most compelling illusory effects (e.g., bilabial phonemes presented acoustically and paired with non-labials, especially alveolar-nasal and velar-occlusive phonemes). In this study EEG was recorded from 128 scalp sites while participants observed a female and a male actor uttering 288 syllables selected on the basis of the previous investigation (lasting approximately 600 ms) and responded to rare targets (/re/, /ri/, /ro/, /ru/). In half of the cases the AV information was incongruent, except for targets that were always congruent. A pMMN (phonological Mismatch Negativity) to incongruent AV stimuli was identified 500 ms after voice onset time. This automatic response indexed the detection of an incongruity between the labial and phonetic information. SwLORETA (Low-Resolution Electromagnetic Tomography) analysis applied to the difference voltage incongruent-congruent in the same time window revealed that the strongest sources of this activity were the right superior temporal (STG) and superior frontal gyri, which supports their involvement in AV integration.
Collapse
Affiliation(s)
- Alice Mado Proverbio
- Neuro-Mi Center for Neuroscience, Dept. of Psychology, University of Milano-Bicocca, Italy.
| | - Giulia Raso
- Neuro-Mi Center for Neuroscience, Dept. of Psychology, University of Milano-Bicocca, Italy
| | | |
Collapse
|
41
|
Stevenson RA, Sheffield SW, Butera IM, Gifford RH, Wallace MT. Multisensory Integration in Cochlear Implant Recipients. Ear Hear 2018; 38:521-538. [PMID: 28399064 DOI: 10.1097/aud.0000000000000435] [Citation(s) in RCA: 53] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
Speech perception is inherently a multisensory process involving integration of auditory and visual cues. Multisensory integration in cochlear implant (CI) recipients is a unique circumstance in that the integration occurs after auditory deprivation and the provision of hearing via the CI. Despite the clear importance of multisensory cues for perception, in general, and for speech intelligibility, specifically, the topic of multisensory perceptual benefits in CI users has only recently begun to emerge as an area of inquiry. We review the research that has been conducted on multisensory integration in CI users to date and suggest a number of areas needing further research. The overall pattern of results indicates that many CI recipients show at least some perceptual gain that can be attributable to multisensory integration. The extent of this gain, however, varies based on a number of factors, including age of implantation and specific task being assessed (e.g., stimulus detection, phoneme perception, word recognition). Although both children and adults with CIs obtain audiovisual benefits for phoneme, word, and sentence stimuli, neither group shows demonstrable gain for suprasegmental feature perception. Additionally, only early-implanted children and the highest performing adults obtain audiovisual integration benefits similar to individuals with normal hearing. Increasing age of implantation in children is associated with poorer gains resultant from audiovisual integration, suggesting a sensitive period in development for the brain networks that subserve these integrative functions, as well as length of auditory experience. This finding highlights the need for early detection of and intervention for hearing loss, not only in terms of auditory perception, but also in terms of the behavioral and perceptual benefits of audiovisual processing. Importantly, patterns of auditory, visual, and audiovisual responses suggest that underlying integrative processes may be fundamentally different between CI users and typical-hearing listeners. Future research, particularly in low-level processing tasks such as signal detection will help to further assess mechanisms of multisensory integration for individuals with hearing loss, both with and without CIs.
Collapse
Affiliation(s)
- Ryan A Stevenson
- 1Department of Psychology, University of Western Ontario, London, Ontario, Canada; 2Brain and Mind Institute, University of Western Ontario, London, Ontario, Canada; 3Walter Reed National Military Medical Center, Audiology and Speech Pathology Center, London, Ontario, Canada; 4Vanderbilt Brain Institute, Nashville, Tennesse; 5Vanderbilt Kennedy Center, Nashville, Tennesse; 6Department of Psychology, Vanderbilt University, Nashville, Tennesse; 7Department of Psychiatry, Vanderbilt University Medical Center, Nashville, Tennesse; and 8Department of Hearing and Speech Sciences, Vanderbilt University Medical Center, Nashville, Tennesse
| | | | | | | | | |
Collapse
|
42
|
Ma Q, Zhang T, Zanetti MV, Shen H, Satterthwaite TD, Wolf DH, Gur RE, Fan Y, Hu D, Busatto GF, Davatzikos C. Classification of multi-site MR images in the presence of heterogeneity using multi-task learning. Neuroimage Clin 2018; 19:476-486. [PMID: 29984156 PMCID: PMC6029565 DOI: 10.1016/j.nicl.2018.04.037] [Citation(s) in RCA: 21] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2017] [Revised: 04/09/2018] [Accepted: 04/28/2018] [Indexed: 12/21/2022]
Abstract
With the advent of Big Data Imaging Analytics applied to neuroimaging, datasets from multiple sites need to be pooled into larger samples. However, heterogeneity across different scanners, protocols and populations, renders the task of finding underlying disease signatures challenging. The current work investigates the value of multi-task learning in finding disease signatures that generalize across studies and populations. Herein, we present a multi-task learning type of formulation, in which different tasks are from different studies and populations being pooled together. We test this approach in an MRI study of the neuroanatomy of schizophrenia (SCZ) by pooling data from 3 different sites and populations: Philadelphia, Sao Paulo and Tianjin (50 controls and 50 patients from each site), which posed integration challenges due to variability in disease chronicity, treatment exposure, and data collection. Some existing methods are also tested for comparison purposes. Experiments show that classification accuracy of multi-site data outperformed that of single-site data and pooled data using multi-task feature learning, and also outperformed other comparison methods. Several anatomical regions were identified to be common discriminant features across sites. These included prefrontal, superior temporal, insular, anterior cingulate cortex, temporo-limbic and striatal regions consistently implicated in the pathophysiology of schizophrenia, as well as the cerebellum, precuneus, and fusiform, middle temporal, inferior parietal, postcentral, angular, lingual and middle occipital gyri. These results indicate that the proposed multi-task learning method is robust in finding consistent and reliable structural brain abnormalities associated with SCZ across different sites, in the presence of multiple sources of heterogeneity.
Collapse
Affiliation(s)
- Qiongmin Ma
- College of Mechatronics and Automation, National University of Defense Technology, Changsha, Hunan 410073, China; Center for Biomedical Image Computing and Analytics, and Department of Radiology, University of Pennsylvania, Philadelphia, PA 19104, United States; Beijing Institute of System Engineering, China.
| | - Tianhao Zhang
- Center for Biomedical Image Computing and Analytics, and Department of Radiology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Marcus V Zanetti
- Laboratory of Psychiatric Neuroimaging (LIM-21), Department and Institute of Psychiatry, Faculty of Medicine, University of São Paulo, São Paulo, Brazil
| | - Hui Shen
- College of Mechatronics and Automation, National University of Defense Technology, Changsha, Hunan 410073, China
| | | | - Daniel H Wolf
- Department of Psychiatry, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Raquel E Gur
- Department of Psychiatry, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Yong Fan
- Center for Biomedical Image Computing and Analytics, and Department of Radiology, University of Pennsylvania, Philadelphia, PA 19104, United States
| | - Dewen Hu
- College of Mechatronics and Automation, National University of Defense Technology, Changsha, Hunan 410073, China
| | - Geraldo F Busatto
- Laboratory of Psychiatric Neuroimaging (LIM-21), Department and Institute of Psychiatry, Faculty of Medicine, University of São Paulo, São Paulo, Brazil
| | - Christos Davatzikos
- Center for Biomedical Image Computing and Analytics, and Department of Radiology, University of Pennsylvania, Philadelphia, PA 19104, United States
| |
Collapse
|
43
|
Abstract
The fact that the face is a source of diverse social signals allows us to use face and person perception as a model system for asking important psychological questions about how our brains are organised. A key issue concerns whether we rely primarily on some form of generic representation of the common physical source of these social signals (the face) to interpret them, or instead create multiple representations by assigning different aspects of the task to different specialist components. Variants of the specialist components hypothesis have formed the dominant theoretical perspective on face perception for more than three decades, but despite this dominance of formally and informally expressed theories, the underlying principles and extent of any division of labour remain uncertain. Here, I discuss three important sources of constraint: first, the evolved structure of the brain; second, the need to optimise responses to different everyday tasks; and third, the statistical structure of faces in the perceiver's environment. I show how these constraints interact to determine the underlying functional organisation of face and person perception.
Collapse
|
44
|
Corporaal SHA, Bruijn SM, Hoogkamer W, Chalavi S, Boisgontier MP, Duysens J, Swinnen SP, Gooijers J. Different neural substrates for precision stepping and fast online step adjustments in youth. Brain Struct Funct 2018; 223:2039-2053. [PMID: 29368052 PMCID: PMC5884917 DOI: 10.1007/s00429-017-1586-9] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2017] [Accepted: 11/30/2017] [Indexed: 12/27/2022]
Abstract
Humans can navigate through challenging environments (e.g., cluttered or uneven terrains) by modifying their preferred gait pattern (e.g., step length, step width, or speed). Growing behavioral and neuroimaging evidence suggests that the ability to modify preferred step patterns requires the recruitment of cognitive resources. In children, it is argued that prolonged development of complex gait is related to the ongoing development of involved brain regions, but this has not been directly investigated yet. Here, we aimed to elucidate the relationship between structural brain properties and complex gait in youth aged 9–18 years. We used volumetric analyses of cortical grey matter (GM) and whole-brain voxelwise statistical analyses of white matter (WM), and utilized a treadmill-based precision stepping task to investigate complex gait. Moreover, precision stepping was performed on step targets which were either unperturbed or perturbed (i.e., unexpectedly shifting to a new location). Our main findings revealed that larger unperturbed precision step error was associated with decreased WM microstructural organization of tracts that are particularly associated with attentional and visual processing functions. These results strengthen the hypothesis that precision stepping on unperturbed step targets is driven by cortical processes. In contrast, no significant correlations were found between perturbed precision stepping and cortical structures, indicating that other (neural) mechanisms may be more important for this type of stepping.
Collapse
Affiliation(s)
- Sharissa H A Corporaal
- Movement Control and Neuroplasticity Research Group, Department of Movement Sciences, KU Leuven, Tervuursevest 101, box 1501, 3001, Leuven, Belgium
| | - Sjoerd M Bruijn
- Movement Control and Neuroplasticity Research Group, Department of Movement Sciences, KU Leuven, Tervuursevest 101, box 1501, 3001, Leuven, Belgium
- Department of Human Movement Sciences, MOVE Research Institute Amsterdam, VU University Amsterdam, Amsterdam, The Netherlands
| | - Wouter Hoogkamer
- Movement Control and Neuroplasticity Research Group, Department of Movement Sciences, KU Leuven, Tervuursevest 101, box 1501, 3001, Leuven, Belgium
- Department of Integrative Physiology, University of Colorado, Boulder, USA
| | - Sima Chalavi
- Movement Control and Neuroplasticity Research Group, Department of Movement Sciences, KU Leuven, Tervuursevest 101, box 1501, 3001, Leuven, Belgium
| | - Matthieu P Boisgontier
- Movement Control and Neuroplasticity Research Group, Department of Movement Sciences, KU Leuven, Tervuursevest 101, box 1501, 3001, Leuven, Belgium
| | - Jacques Duysens
- Movement Control and Neuroplasticity Research Group, Department of Movement Sciences, KU Leuven, Tervuursevest 101, box 1501, 3001, Leuven, Belgium
| | - Stephan P Swinnen
- Movement Control and Neuroplasticity Research Group, Department of Movement Sciences, KU Leuven, Tervuursevest 101, box 1501, 3001, Leuven, Belgium
- Leuven Research Institute for Neuroscience and Disease (LIND), KU Leuven, Leuven, Belgium
| | - Jolien Gooijers
- Movement Control and Neuroplasticity Research Group, Department of Movement Sciences, KU Leuven, Tervuursevest 101, box 1501, 3001, Leuven, Belgium.
| |
Collapse
|
45
|
Alsius A, Paré M, Munhall KG. Forty Years After Hearing Lips and Seeing Voices: the McGurk Effect Revisited. Multisens Res 2018; 31:111-144. [PMID: 31264597 DOI: 10.1163/22134808-00002565] [Citation(s) in RCA: 52] [Impact Index Per Article: 8.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2016] [Accepted: 03/09/2017] [Indexed: 11/19/2022]
Abstract
Since its discovery 40 years ago, the McGurk illusion has been usually cited as a prototypical paradigmatic case of multisensory binding in humans, and has been extensively used in speech perception studies as a proxy measure for audiovisual integration mechanisms. Despite the well-established practice of using the McGurk illusion as a tool for studying the mechanisms underlying audiovisual speech integration, the magnitude of the illusion varies enormously across studies. Furthermore, the processing of McGurk stimuli differs from congruent audiovisual processing at both phenomenological and neural levels. This questions the suitability of this illusion as a tool to quantify the necessary and sufficient conditions under which audiovisual integration occurs in natural conditions. In this paper, we review some of the practical and theoretical issues related to the use of the McGurk illusion as an experimental paradigm. We believe that, without a richer understanding of the mechanisms involved in the processing of the McGurk effect, experimenters should be really cautious when generalizing data generated by McGurk stimuli to matching audiovisual speech events.
Collapse
Affiliation(s)
- Agnès Alsius
- Psychology Department, Queen's University, Humphrey Hall, 62 Arch St., Kingston, Ontario, K7L 3N6 Canada
| | - Martin Paré
- Psychology Department, Queen's University, Humphrey Hall, 62 Arch St., Kingston, Ontario, K7L 3N6 Canada
| | - Kevin G Munhall
- Psychology Department, Queen's University, Humphrey Hall, 62 Arch St., Kingston, Ontario, K7L 3N6 Canada
| |
Collapse
|
46
|
Shen G, Meltzoff AN, Marshall PJ. Touching lips and hearing fingers: effector-specific congruency between tactile and auditory stimulation modulates N1 amplitude and alpha desynchronization. Exp Brain Res 2018; 236:13-29. [PMID: 29038847 PMCID: PMC5976883 DOI: 10.1007/s00221-017-5104-3] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2017] [Accepted: 10/05/2017] [Indexed: 10/18/2022]
Abstract
Understanding the interactions between audition and sensorimotor processes is of theoretical importance, particularly in relation to speech processing. Although one current focus in this area is on interactions between auditory perception and the motor system, there has been less research on connections between the auditory and somatosensory modalities. The current study takes a novel approach to this omission by examining specific auditory-tactile interactions in the context of speech and non-speech sound production. Electroencephalography was used to examine brain responses when participants were presented with speech syllables (a bilabial sound /pa/ and a non-labial sound /ka/) or finger-snapping sounds that were simultaneously paired with tactile stimulation of either the lower lip or the right middle finger. Analyses focused on the sensory-evoked N1 in the event-related potential and the extent of alpha band desynchronization elicited by the stimuli. N1 amplitude over fronto-central sites was significantly enhanced when the bilabial /pa/ sound was paired with tactile lip stimulation and when the finger-snapping sound was paired with tactile stimulation of the finger. Post-stimulus alpha desynchronization at central sites was also enhanced when the /pa/ sound was accompanied by tactile stimulation of the lip. These novel findings indicate that neural aspects of somatosensory-auditory interactions are influenced by the congruency between the location of the bodily touch and the bodily origin of a perceived sound.
Collapse
Affiliation(s)
- Guannan Shen
- Department of Psychology, 1701 N 13th Street, Philadelphia, PA, 19122, USA.
| | - Andrew N Meltzoff
- Department of Psychology, 1701 N 13th Street, Philadelphia, PA, 19122, USA
- Institute for Learning and Brian Sciences, University of Washington, Seattle, USA
| | - Peter J Marshall
- Department of Psychology, 1701 N 13th Street, Philadelphia, PA, 19122, USA
| |
Collapse
|
47
|
Giordano BL, Ince RAA, Gross J, Schyns PG, Panzeri S, Kayser C. Contributions of local speech encoding and functional connectivity to audio-visual speech perception. eLife 2017; 6. [PMID: 28590903 PMCID: PMC5462535 DOI: 10.7554/elife.24763] [Citation(s) in RCA: 43] [Impact Index Per Article: 6.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2016] [Accepted: 05/07/2017] [Indexed: 11/13/2022] Open
Abstract
Seeing a speaker’s face enhances speech intelligibility in adverse environments. We investigated the underlying network mechanisms by quantifying local speech representations and directed connectivity in MEG data obtained while human participants listened to speech of varying acoustic SNR and visual context. During high acoustic SNR speech encoding by temporally entrained brain activity was strong in temporal and inferior frontal cortex, while during low SNR strong entrainment emerged in premotor and superior frontal cortex. These changes in local encoding were accompanied by changes in directed connectivity along the ventral stream and the auditory-premotor axis. Importantly, the behavioral benefit arising from seeing the speaker’s face was not predicted by changes in local encoding but rather by enhanced functional connectivity between temporal and inferior frontal cortex. Our results demonstrate a role of auditory-frontal interactions in visual speech representations and suggest that functional connectivity along the ventral pathway facilitates speech comprehension in multisensory environments. DOI:http://dx.doi.org/10.7554/eLife.24763.001 When listening to someone in a noisy environment, such as a cocktail party, we can understand the speaker more easily if we can also see his or her face. Movements of the lips and tongue convey additional information that helps the listener’s brain separate out syllables, words and sentences. However, exactly where in the brain this effect occurs and how it works remain unclear. To find out, Giordano et al. scanned the brains of healthy volunteers as they watched clips of people speaking. The clarity of the speech varied between clips. Furthermore, in some of the clips the lip movements of the speaker corresponded to the speech in question, whereas in others the lip movements were nonsense babble. As expected, the volunteers performed better on a word recognition task when the speech was clear and when the lips movements agreed with the spoken dialogue. Watching the video clips stimulated rhythmic activity in multiple regions of the volunteers’ brains, including areas that process sound and areas that plan movements. Speech is itself rhythmic, and the volunteers’ brain activity synchronized with the rhythms of the speech they were listening to. Seeing the speaker’s face increased this degree of synchrony. However, it also made it easier for sound-processing regions within the listeners’ brains to transfer information to one other. Notably, only the latter effect predicted improved performance on the word recognition task. This suggests that seeing a person’s face makes it easier to understand his or her speech by boosting communication between brain regions, rather than through effects on individual areas. Further work is required to determine where and how the brain encodes lip movements and speech sounds. The next challenge will be to identify where these two sets of information interact, and how the brain merges them together to generate the impression of specific words. DOI:http://dx.doi.org/10.7554/eLife.24763.002
Collapse
Affiliation(s)
- Bruno L Giordano
- Institut de Neurosciences de la Timone UMR 7289, Aix Marseille Université - Centre National de la Recherche Scientifique, Marseille, France.,Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| | - Robin A A Ince
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| | - Joachim Gross
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| | - Philippe G Schyns
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| | - Stefano Panzeri
- Neural Computation Laboratory, Center for Neuroscience and Cognitive Systems, Istituto Italiano di Tecnologia, Rovereto, Italy
| | - Christoph Kayser
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom
| |
Collapse
|
48
|
|
49
|
Venezia JH, Vaden KI, Rong F, Maddox D, Saberi K, Hickok G. Auditory, Visual and Audiovisual Speech Processing Streams in Superior Temporal Sulcus. Front Hum Neurosci 2017; 11:174. [PMID: 28439236 PMCID: PMC5383672 DOI: 10.3389/fnhum.2017.00174] [Citation(s) in RCA: 37] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2016] [Accepted: 03/24/2017] [Indexed: 11/30/2022] Open
Abstract
The human superior temporal sulcus (STS) is responsive to visual and auditory information, including sounds and facial cues during speech recognition. We investigated the functional organization of STS with respect to modality-specific and multimodal speech representations. Twenty younger adult participants were instructed to perform an oddball detection task and were presented with auditory, visual, and audiovisual speech stimuli, as well as auditory and visual nonspeech control stimuli in a block fMRI design. Consistent with a hypothesized anterior-posterior processing gradient in STS, auditory, visual and audiovisual stimuli produced the largest BOLD effects in anterior, posterior and middle STS (mSTS), respectively, based on whole-brain, linear mixed effects and principal component analyses. Notably, the mSTS exhibited preferential responses to multisensory stimulation, as well as speech compared to nonspeech. Within the mid-posterior and mSTS regions, response preferences changed gradually from visual, to multisensory, to auditory moving posterior to anterior. Post hoc analysis of visual regions in the posterior STS revealed that a single subregion bordering the mSTS was insensitive to differences in low-level motion kinematics yet distinguished between visual speech and nonspeech based on multi-voxel activation patterns. These results suggest that auditory and visual speech representations are elaborated gradually within anterior and posterior processing streams, respectively, and may be integrated within the mSTS, which is sensitive to more abstract speech information within and across presentation modalities. The spatial organization of STS is consistent with processing streams that are hypothesized to synthesize perceptual speech representations from sensory signals that provide convergent information from visual and auditory modalities.
Collapse
Affiliation(s)
| | - Kenneth I Vaden
- Department of Otolaryngology-Head and Neck Surgery, Medical University of South CarolinaCharleston, SC, USA
| | - Feng Rong
- Department of Cognitive Sciences, Center for Cognitive Neuroscience and Engineering, University of CaliforniaIrvine, CA, USA
| | - Dale Maddox
- Department of Cognitive Sciences, Center for Cognitive Neuroscience and Engineering, University of CaliforniaIrvine, CA, USA
| | - Kourosh Saberi
- Department of Cognitive Sciences, Center for Cognitive Neuroscience and Engineering, University of CaliforniaIrvine, CA, USA
| | - Gregory Hickok
- Department of Cognitive Sciences, Center for Cognitive Neuroscience and Engineering, University of CaliforniaIrvine, CA, USA
| |
Collapse
|
50
|
Audio-visual speech perception in adult readers with dyslexia: an fMRI study. Brain Imaging Behav 2017; 12:357-368. [DOI: 10.1007/s11682-017-9694-y] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
|