1
|
Silva AB, Liu JR, Metzger SL, Bhaya-Grossman I, Dougherty ME, Seaton MP, Littlejohn KT, Tu-Chan A, Ganguly K, Moses DA, Chang EF. A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages. Nat Biomed Eng 2024; 8:977-991. [PMID: 38769157 DOI: 10.1038/s41551-024-01207-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 04/01/2024] [Indexed: 05/22/2024]
Abstract
Advancements in decoding speech from brain activity have focused on decoding a single language. Hence, the extent to which bilingual speech production relies on unique or shared cortical activity across languages has remained unclear. Here, we leveraged electrocorticography, along with deep-learning and statistical natural-language models of English and Spanish, to record and decode activity from speech-motor cortex of a Spanish-English bilingual with vocal-tract and limb paralysis into sentences in either language. This was achieved without requiring the participant to manually specify the target language. Decoding models relied on shared vocal-tract articulatory representations across languages, which allowed us to build a syllable classifier that generalized across a shared set of English and Spanish syllables. Transfer learning expedited training of the bilingual decoder by enabling neural data recorded in one language to improve decoding in the other language. Overall, our findings suggest shared cortical articulatory representations that persist after paralysis and enable the decoding of multiple languages without the need to train separate language-specific decoders.
Collapse
Affiliation(s)
- Alexander B Silva
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Jessie R Liu
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Sean L Metzger
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Ilina Bhaya-Grossman
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA
| | - Maximilian E Dougherty
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
| | - Margaret P Seaton
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
| | - Kaylo T Littlejohn
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA, USA
| | - Adelyn Tu-Chan
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - Karunesh Ganguly
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
- Department of Neurology, University of California, San Francisco, San Francisco, CA, USA
| | - David A Moses
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA
| | - Edward F Chang
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA, USA.
- Weill Institute for Neuroscience, University of California, San Francisco, San Francisco, CA, USA.
- University of California, Berkeley - University of California, San Francisco Graduate Program in Bioengineering, Berkeley, CA, USA.
| |
Collapse
|
2
|
Functional Hemispheric Activity and Asymmetry Markers of Effective Foreign Language Performance in 3rd-Grade, 10th-Grade, and University Students. Symmetry (Basel) 2022. [DOI: 10.3390/sym14081659] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
Abstract
The activity of the left hemisphere is often associated with linguistic functioning, including in a foreign language. At the same time, research results demonstrate that different structures in both hemispheres can be jointly activated in the performance of particular linguistic tasks. The current study aimed to identify functional hemispheric activity and asymmetry markers for effective foreign language performance. The study sample consisted of 27 3rd-grade, 26 10th-grade, and 21 university students, all native Russian. To measure functional hemispheric asymmetry and activity before and after an English class and before an English test, we used computer laterometry in the ‘two-source’ lead–lag dichotic paradigm. The study results reveal that left hemispheric functional dominance can be considered as a marker for effective activity during an English class and an English test in 3rd-grade and 10th-grade students. In university students, right hemispheric functional dominance predicted better efficacy during the English class. Therefore, the results obtained provide evidence about different hemispheric activity and asymmetry modes for different ages of foreign language mastering, and the results may support the hypothesis about the possibility of a ‘sensitive period’ for foreign language acquisition occurring at any age. These findings can be applied to the creation of biofeedback trainings for hemispheric profile optimization when learning a foreign language and may help in creating personalized learning schedules.
Collapse
|
3
|
Treutler M, Sörös P. Functional MRI of Native and Non-native Speech Sound Production in Sequential German-English Bilinguals. Front Hum Neurosci 2021; 15:683277. [PMID: 34349632 PMCID: PMC8326338 DOI: 10.3389/fnhum.2021.683277] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2021] [Accepted: 06/22/2021] [Indexed: 11/13/2022] Open
Abstract
Bilingualism and multilingualism are highly prevalent. Non-invasive brain imaging has been used to study the neural correlates of native and non-native speech and language production, mainly on the lexical and syntactic level. Here, we acquired continuous fast event-related FMRI during visually cued overt production of exclusively German and English vowels and syllables. We analyzed data from 13 university students, native speakers of German and sequential English bilinguals. The production of non-native English sounds was associated with increased activity of the left primary sensorimotor cortex, bilateral cerebellar hemispheres (lobule VI), left inferior frontal gyrus, and left anterior insula compared to native German sounds. The contrast German > English sounds was not statistically significant. Our results emphasize that the production of non-native speech requires additional neural resources already on a basic phonological level in sequential bilinguals.
Collapse
Affiliation(s)
- Miriam Treutler
- European Medical School Oldenburg-Groningen, Carl von Ossietzky University of Oldenburg, Oldenburg, Germany
| | - Peter Sörös
- Department of Neurology, Carl von Ossietzky University of Oldenburg, Oldenburg, Germany.,Research Center Neurosensory Science, Carl von Ossietzky University of Oldenburg, Oldenburg, Germany
| |
Collapse
|
4
|
Kasaba R, Shimada K, Tomoda A. Neural Mechanisms of Parental Communicative Adjustments in Spoken Language. Neuroscience 2020; 457:206-217. [PMID: 33346117 DOI: 10.1016/j.neuroscience.2020.12.002] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2020] [Revised: 12/01/2020] [Accepted: 12/04/2020] [Indexed: 11/30/2022]
Abstract
During cultural transmission, caregivers typically adjust their form of speech according to the presumed characteristics of an infant/child, a phenomenon known as infant/child directed speech (IDS/CDS) or "parentese." Although ventromedial prefrontal cortex (vmPFC) damage was previously found to be associated with failure in adjusting non-verbal communicative behaviors, little is known about the neural mechanisms of verbal communicative adjustments, such as IDS/CDS. In the current study, 30 healthy mothers with preschool-age children underwent functional magnetic resonance imaging (fMRI) while performing a picture naming task which required them to name an object for either a child or an adult. In the picture naming task, mothers exhibited a longer naming duration in the toward-child condition than the toward-adult control condition. Naming an object for a child, compared with naming it for an adult, resulted in greater involvement in the vmPFC and other regions (e.g., cerebellum) in the global caregiving network. In particular, the vmPFC exhibited task-related deactivation and decreased functional connectivity with the supplementary motor, precentral, postcentral, and supramarginal regions. These findings suggest that the vmPFC, which is included in the default mode network, is involved in optimizing communicative behaviors for the inter-generational transmission of knowledge. This function of the vmPFC may be considered as a prosocial drive to lead to prosocial communicative behaviors depending on the context. This study provides a better understanding of the neural mechanisms involved in communicative adjustments for children and insight into related applied research fields such as parenting, pedagogy, and education.
Collapse
Affiliation(s)
- Ryoko Kasaba
- Division of Developmental Higher Brain Functions, United Graduate School of Child Development, University of Fukui, 23-3 Matsuoka-Shimoaizuki, Eiheiji-cho, Yoshida-gun, Fukui 910-1193, Japan
| | - Koji Shimada
- Division of Developmental Higher Brain Functions, United Graduate School of Child Development, University of Fukui, 23-3 Matsuoka-Shimoaizuki, Eiheiji-cho, Yoshida-gun, Fukui 910-1193, Japan; Research Center for Child Mental Development, University of Fukui, 23-3 Matsuoka-Shimoaizuki, Eiheiji-cho, Yoshida-gun, Fukui 910-1193, Japan; Biomedical Imaging Research Center, University of Fukui, 23-3 Matsuoka-Shimoaizuki,Eiheiji-cho, Yoshida-gun, Fukui 910-1193, Japan.
| | - Akemi Tomoda
- Division of Developmental Higher Brain Functions, United Graduate School of Child Development, University of Fukui, 23-3 Matsuoka-Shimoaizuki, Eiheiji-cho, Yoshida-gun, Fukui 910-1193, Japan; Research Center for Child Mental Development, University of Fukui, 23-3 Matsuoka-Shimoaizuki, Eiheiji-cho, Yoshida-gun, Fukui 910-1193, Japan; Department of Child and Adolescent Psychological Medicine, University of Fukui Hospital, 23-3 Matsuoka-Shimoaizuki, Eiheiji-cho, Yoshida-gun, Fukui 910-1193, Japan.
| |
Collapse
|
5
|
Glanz Iljina O, Derix J, Kaur R, Schulze-Bonhage A, Auer P, Aertsen A, Ball T. Real-life speech production and perception have a shared premotor-cortical substrate. Sci Rep 2018; 8:8898. [PMID: 29891885 PMCID: PMC5995900 DOI: 10.1038/s41598-018-26801-x] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2017] [Accepted: 05/09/2018] [Indexed: 11/25/2022] Open
Abstract
Motor-cognitive accounts assume that the articulatory cortex is involved in language comprehension, but previous studies may have observed such an involvement as an artefact of experimental procedures. Here, we employed electrocorticography (ECoG) during natural, non-experimental behavior combined with electrocortical stimulation mapping to study the neural basis of real-life human verbal communication. We took advantage of ECoG’s ability to capture high-gamma activity (70–350 Hz) as a spatially and temporally precise index of cortical activation during unconstrained, naturalistic speech production and perception conditions. Our findings show that an electrostimulation-defined mouth motor region located in the superior ventral premotor cortex is consistently activated during both conditions. This region became active early relative to the onset of speech production and was recruited during speech perception regardless of acoustic background noise. Our study thus pinpoints a shared ventral premotor substrate for real-life speech production and perception with its basic properties.
Collapse
Affiliation(s)
- Olga Glanz Iljina
- GRK 1624 'Frequency Effects in Language', University of Freiburg, Freiburg, Germany. .,Department of German Linguistics, University of Freiburg, Freiburg, Germany. .,Hermann Paul School of Linguistics, University of Freiburg, Freiburg, Germany. .,Translational Neurotechnology Lab, Department of Neurosurgery, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany. .,BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany. .,Neurobiology and Biophysics, Faculty of Biology, University of Freiburg, Freiburg, Germany.
| | - Johanna Derix
- Translational Neurotechnology Lab, Department of Neurosurgery, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany.,BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany.,Neurobiology and Biophysics, Faculty of Biology, University of Freiburg, Freiburg, Germany
| | - Rajbir Kaur
- Translational Neurotechnology Lab, Department of Neurosurgery, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany.,Faculty of Medicine, University of Cologne, Cologne, Germany
| | - Andreas Schulze-Bonhage
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany.,Epilepsy Center, Department of Neurosurgery, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany.,Bernstein Center Freiburg, University of Freiburg, Freiburg, Germany
| | - Peter Auer
- GRK 1624 'Frequency Effects in Language', University of Freiburg, Freiburg, Germany.,Department of German Linguistics, University of Freiburg, Freiburg, Germany.,Hermann Paul School of Linguistics, University of Freiburg, Freiburg, Germany
| | - Ad Aertsen
- Neurobiology and Biophysics, Faculty of Biology, University of Freiburg, Freiburg, Germany.,Bernstein Center Freiburg, University of Freiburg, Freiburg, Germany
| | - Tonio Ball
- Translational Neurotechnology Lab, Department of Neurosurgery, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany. .,BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany. .,Bernstein Center Freiburg, University of Freiburg, Freiburg, Germany.
| |
Collapse
|