1
|
Karthik G, Cao CZ, Demidenko MI, Jahn A, Stacey WC, Wasade VS, Brang D. Auditory cortex encodes lipreading information through spatially distributed activity. Curr Biol 2024; 34:4021-4032.e5. [PMID: 39153482 DOI: 10.1016/j.cub.2024.07.073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Revised: 04/29/2024] [Accepted: 07/19/2024] [Indexed: 08/19/2024]
Abstract
Watching a speaker's face improves speech perception accuracy. This benefit is enabled, in part, by implicit lipreading abilities present in the general population. While it is established that lipreading can alter the perception of a heard word, it is unknown how these visual signals are represented in the auditory system or how they interact with auditory speech representations. One influential, but untested, hypothesis is that visual speech modulates the population-coded representations of phonetic and phonemic features in the auditory system. This model is largely supported by data showing that silent lipreading evokes activity in the auditory cortex, but these activations could alternatively reflect general effects of arousal or attention or the encoding of non-linguistic features such as visual timing information. This gap limits our understanding of how vision supports speech perception. To test the hypothesis that the auditory system encodes visual speech information, we acquired functional magnetic resonance imaging (fMRI) data from healthy adults and intracranial recordings from electrodes implanted in patients with epilepsy during auditory and visual speech perception tasks. Across both datasets, linear classifiers successfully decoded the identity of silently lipread words using the spatial pattern of auditory cortex responses. Examining the time course of classification using intracranial recordings, lipread words were classified at earlier time points relative to heard words, suggesting a predictive mechanism for facilitating speech. These results support a model in which the auditory system combines the joint neural distributions evoked by heard and lipread words to generate a more precise estimate of what was said.
Collapse
Affiliation(s)
- Ganesan Karthik
- Department of Psychology, University of Michigan, Ann Arbor, MI 48109, USA
| | - Cody Zhewei Cao
- Department of Psychology, University of Michigan, Ann Arbor, MI 48109, USA
| | | | - Andrew Jahn
- Department of Psychology, University of Michigan, Ann Arbor, MI 48109, USA
| | - William C Stacey
- Department of Neurology, University of Michigan, Ann Arbor, MI 48109, USA
| | - Vibhangini S Wasade
- Henry Ford Hospital, Detroit, MI 48202, USA; Department of Neurology, Wayne State University School of Medicine, Detroit, MI 48201, USA
| | - David Brang
- Department of Psychology, University of Michigan, Ann Arbor, MI 48109, USA.
| |
Collapse
|
2
|
Khan H, Khadka R, Sultan MS, Yazidi A, Ombao H, Mirtaheri P. Unleashing the potential of fNIRS with machine learning: classification of fine anatomical movements to empower future brain-computer interface. Front Hum Neurosci 2024; 18:1354143. [PMID: 38435744 PMCID: PMC10904609 DOI: 10.3389/fnhum.2024.1354143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/12/2023] [Accepted: 01/31/2024] [Indexed: 03/05/2024] Open
Abstract
In this study, we explore the potential of using functional near-infrared spectroscopy (fNIRS) signals in conjunction with modern machine-learning techniques to classify specific anatomical movements to increase the number of control commands for a possible fNIRS-based brain-computer interface (BCI) applications. The study focuses on novel individual finger-tapping, a well-known task in fNIRS and fMRI studies, but limited to left/right or few fingers. Twenty-four right-handed participants performed the individual finger-tapping task. Data were recorded by using sixteen sources and detectors placed over the motor cortex according to the 10-10 international system. The event's average oxygenated Δ HbO and deoxygenated Δ HbR hemoglobin data were utilized as features to assess the performance of diverse machine learning (ML) models in a challenging multi-class classification setting. These methods include LDA, QDA, MNLR, XGBoost, and RF. A new DL-based model named "Hemo-Net" has been proposed which consists of multiple parallel convolution layers with different filters to extract the features. This paper aims to explore the efficacy of using fNRIS along with ML/DL methods in a multi-class classification task. Complex models like RF, XGBoost, and Hemo-Net produce relatively higher test set accuracy when compared to LDA, MNLR, and QDA. Hemo-Net has depicted a superior performance achieving the highest test set accuracy of 76%, however, in this work, we do not aim at improving the accuracies of models rather we are interested in exploring if fNIRS has the neural signatures to help modern ML/DL methods in multi-class classification which can lead to applications like brain-computer interfaces. Multi-class classification of fine anatomical movements, such as individual finger movements, is difficult to classify with fNIRS data. Traditional ML models like MNLR and LDA show inferior performance compared to the ensemble-based methods of RF and XGBoost. DL-based method Hemo-Net outperforms all methods evaluated in this study and demonstrates a promising future for fNIRS-based BCI applications.
Collapse
Affiliation(s)
- Haroon Khan
- Department of Mechanical, Electronics and Chemical Engineering, OsloMet - Oslo Metropolitan University, Oslo, Norway
| | - Rabindra Khadka
- Department of Information Technology, Oslomet - Oslo Metropolitan University, Oslo, Norway
| | - Malik Shahid Sultan
- Department of Computer, Electrical and Mathematical Science and Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
| | - Anis Yazidi
- Department of Information Technology, Oslomet - Oslo Metropolitan University, Oslo, Norway
| | - Hernando Ombao
- Department of Computer, Electrical and Mathematical Science and Engineering, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia
| | - Peyman Mirtaheri
- Department of Mechanical, Electronics and Chemical Engineering, OsloMet - Oslo Metropolitan University, Oslo, Norway
| |
Collapse
|
3
|
Barany DA, Lacey S, Matthews KL, Nygaard LC, Sathian K. Neural basis of sound-symbolic pseudoword-shape correspondences. Neuropsychologia 2023; 188:108657. [PMID: 37543139 PMCID: PMC10529692 DOI: 10.1016/j.neuropsychologia.2023.108657] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2023] [Revised: 06/23/2023] [Accepted: 08/02/2023] [Indexed: 08/07/2023]
Abstract
Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.
Collapse
Affiliation(s)
- Deborah A Barany
- Department of Kinesiology, University of Georgia and Augusta University/University of Georgia Medical Partnership, Athens, GA, 30602, USA
| | - Simon Lacey
- Department of Neurology, Penn State College of Medicine, Hershey, PA, 17033-0859, USA; Department of Neural & Behavioral Sciences, Penn State College of Medicine, Hershey, PA, 17033-0859, USA; Department of Psychology, Penn State College of Liberal Arts, University Park, PA, 16802, USA
| | - Kaitlyn L Matthews
- Department of Psychology, Emory University, Atlanta, GA, 30322, USA; Present address: Department of Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, MO, 63130, USA
| | - Lynne C Nygaard
- Department of Psychology, Emory University, Atlanta, GA, 30322, USA
| | - K Sathian
- Department of Neurology, Penn State College of Medicine, Hershey, PA, 17033-0859, USA; Department of Neural & Behavioral Sciences, Penn State College of Medicine, Hershey, PA, 17033-0859, USA; Department of Psychology, Penn State College of Liberal Arts, University Park, PA, 16802, USA.
| |
Collapse
|
4
|
Kahn AE, Szymula K, Loman S, Haggerty EB, Nyema N, Aguirre GK, Bassett DS. Network structure influences the strength of learned neural representations. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.01.23.525254. [PMID: 36747703 PMCID: PMC9900848 DOI: 10.1101/2023.01.23.525254] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 01/25/2023]
Abstract
Human experience is built upon sequences of discrete events. From those sequences, humans build impressively accurate models of their world. This process has been referred to as graph learning, a form of structure learning in which the mental model encodes the graph of event-to-event transition probabilities [1], [2], typically in medial temporal cortex [3]-[6]. Recent evidence suggests that some network structures are easier to learn than others [7]-[9], but the neural properties of this effect remain unknown. Here we use fMRI to show that the network structure of a temporal sequence of stimuli influences the fidelity with which those stimuli are represented in the brain. Healthy adult human participants learned a set of stimulus-motor associations following one of two graph structures. The design of our experiment allowed us to separate regional sensitivity to the structural, stimulus, and motor response components of the task. As expected, whereas the motor response could be decoded from neural representations in postcentral gyrus, the shape of the stimulus could be decoded from lateral occipital cortex. The structure of the graph impacted the nature of neural representations: when the graph was modular as opposed to lattice-like, BOLD representations in visual areas better predicted trial identity in a held-out run and displayed higher intrinsic dimensionality. Our results demonstrate that even over relatively short timescales, graph structure determines the fidelity of event representations as well as the dimensionality of the space in which those representations are encoded. More broadly, our study shows that network context influences the strength of learned neural representations, motivating future work in the design, optimization, and adaptation of network contexts for distinct types of learning over different timescales.
Collapse
Affiliation(s)
- Ari E. Kahn
- Princeton Neuroscience Institute and Department of Psychology, Princeton University, Princeton, NJ, 08540 USA
| | - Karol Szymula
- Medical Scientist Training Program, University of Rochester School of Medicine and Dentistry, Rochester, New York, 14642 USA
| | - Sophie Loman
- Department of Bioengineering, School of Engineering & Applied Science, University of Pennsylvania, Philadelphia, PA 19104 USA
| | - Edda B. Haggerty
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104 USA
| | - Nathaniel Nyema
- Department of Bioengineering, School of Engineering & Applied Science, University of Pennsylvania, Philadelphia, PA 19104 USA
| | - Geoffrey K. Aguirre
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104 USA
| | - Dani S. Bassett
- Department of Bioengineering, School of Engineering & Applied Science, University of Pennsylvania, Philadelphia, PA 19104 USA
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104 USA
- Department of Electrical & Systems Engineering, School of Engineering & Applied Science, University of Pennsylvania, Philadelphia, PA 19104 USA
- Department of Physics & Astronomy, College of Arts & Sciences, University of Pennsylvania, Philadelphia, PA 19104 USA
- Department of Psychiatry, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19104 USA
- Santa Fe Institute, Santa Fe, NM 87501, USA
| |
Collapse
|
5
|
Barany DA, Lacey S, Matthews KL, Nygaard LC, Sathian K. Neural Basis Of Sound-Symbolic Pseudoword-Shape Correspondences. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.04.14.536865. [PMID: 37425853 PMCID: PMC10327042 DOI: 10.1101/2023.04.14.536865] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/11/2023]
Abstract
Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants ( n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration. HIGHLIGHTS fMRI investigation of sound-symbolic correspondences between auditory pseudowords and visual shapesFaster reaction times for congruent than incongruent audiovisual stimuliGreater activation in auditory and visual cortices for congruent stimuliHigher classification accuracy for congruent stimuli in language and visual areasSound symbolism involves language processing and multisensory integration.
Collapse
Affiliation(s)
- Deborah A. Barany
- Department of Kinesiology, University of Georgia and Augusta University/University of Georgia Medical Partnership, Athens, GA, 30602, USA
| | - Simon Lacey
- Department of Neurology, Penn State Colleges of Medicine and Liberal Arts, Hershey, PA 17033-0859, USA
- Department of Neural & Behavioral Sciences, Penn State Colleges of Medicine and Liberal Arts, Hershey, PA 17033-0859, USA
- Department of Psychology, Penn State Colleges of Medicine and Liberal Arts, Hershey, PA 17033-0859, USA
| | - Kaitlyn L. Matthews
- Department of Psychology, Emory University, Atlanta, GA 30322, USA
- Present address: Department of Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, MO 63130
| | - Lynne C. Nygaard
- Department of Psychology, Emory University, Atlanta, GA 30322, USA
| | - K. Sathian
- Department of Neurology, Penn State Colleges of Medicine and Liberal Arts, Hershey, PA 17033-0859, USA
- Department of Neural & Behavioral Sciences, Penn State Colleges of Medicine and Liberal Arts, Hershey, PA 17033-0859, USA
- Department of Psychology, Penn State Colleges of Medicine and Liberal Arts, Hershey, PA 17033-0859, USA
| |
Collapse
|
6
|
Choupan J, Douglas PK, Gal Y, Cohen MS, Reutens DC, Yang Z. Temporal embedding and spatiotemporal feature selection boost multi-voxel pattern analysis decoding accuracy. J Neurosci Methods 2020; 345:108836. [PMID: 32726664 DOI: 10.1016/j.jneumeth.2020.108836] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2019] [Revised: 06/24/2020] [Accepted: 06/28/2020] [Indexed: 10/23/2022]
Abstract
BACKGROUND In fMRI decoding, temporal embedding of spatial features of the brain allows the incorporation of brain activity dynamics into the multivariate pattern classification process, and provides enriched information about stimulus-specific response patterns and potentially improved prediction accuracy. NEW METHOD This study investigates the possibility of enhancing the classification performance by exploring temporal embedding, to identify the optimum combination of spatiotemporal features based on their classification performance. We investigated the importance of spatiotemporal feature selection using a slow event-related design adapted from the classic Haxby study (Haxby et al., 2001). Data were collected using a multiband fMRI sequence with temporal resolution of 0.568 s. COMPARISON WITH EXISTING METHODS A wide range of spatiotemporal observations were created as various combinations of spatiotemporal features. Using both random forest, and support vector machine, classifiers prediction accuracies for these combinations were then compared with the single spatial multivariate pattern approach that uses only a single temporal observation. RESULTS Our findings showed that, on average, spatiotemporal feature selection improved prediction accuracy. Moreover, the random forest algorithm outperformed the support vector machine and benefitted from temporal information to a greater extent. CONCLUSIONS As expected, the most influential temporal durations were found to be around the peak of the hemodynamic response function, a few seconds after the stimuli onset until -4 s after the peak of the hemodynamic response function. The superiority of spatiotemporal feature selection over single time-point spatial approaches invites future work to design optimal approaches that incorporate spatiotemporal dependencies into feature selection for decoding.
Collapse
Affiliation(s)
- Jeiran Choupan
- Centre for Advanced Imaging, The University of Queensland, Brisbane, Australia; Queensland Brain Institute, The University of Queensland, Brisbane, Australia; Department of Psychology, USC Dornsife College of Letters, Arts and Sciences, University of Southern California, USA; Laboratory of Neuro Imaging, USC Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA.
| | - Pamela K Douglas
- Center for Cognitive Neuroscience, University of California, Los Angeles, CA, USA; Modeling & Simulation, and Computer Science Departments, UCF, Florida, USA
| | - Yaniv Gal
- School of Information Technology and Electrical Engineering, The University of Queensland, Brisbane, Australia
| | - Mark S Cohen
- Neuropsychiatric Institute, University of California, Los Angeles, CA, USA; Departments of Psychiatry and Behavioral Sciences, Neurology, Radiological Sciences, Biomedical Physics, Psychology, and Bioengineering, University of California, Los Angeles, CA, USA; California Nanosystems Institute UCLA School of Medicine, Los Angeles, CA, USA
| | - David C Reutens
- Centre for Advanced Imaging, The University of Queensland, Brisbane, Australia
| | - Zhengyi Yang
- Centre for Advanced Imaging, The University of Queensland, Brisbane, Australia; School of Information Technology and Electrical Engineering, The University of Queensland, Brisbane, Australia; Brainnetome Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
7
|
Li H, Fan Y. Interpretable, highly accurate brain decoding of subtly distinct brain states from functional MRI using intrinsic functional networks and long short-term memory recurrent neural networks. Neuroimage 2019; 202:116059. [PMID: 31362049 PMCID: PMC6819260 DOI: 10.1016/j.neuroimage.2019.116059] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2019] [Revised: 07/25/2019] [Accepted: 07/26/2019] [Indexed: 11/17/2022] Open
Abstract
Decoding brain functional states underlying cognitive processes from functional MRI (fMRI) data using multivariate pattern analysis (MVPA) techniques has achieved promising performance for characterizing brain activation patterns and providing neurofeedback signals. However, it remains challenging to decode subtly distinct brain states for individual fMRI data points due to varying temporal durations and dependency among different cognitive processes. In this study, we develop a deep learning based framework for brain decoding by leveraging recent advances in intrinsic functional network modeling and sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, subject-specific intrinsic functional networks (FNs) are computed from resting-state fMRI data and are used to characterize functional signals of task fMRI data with a compact representation for building brain decoding models, and LSTM RNNs are adopted to learn brain decoding mappings between functional profiles and brain states. Validation results on fMRI data from the HCP dataset have demonstrated that brain decoding models built on training data using the proposed method could learn discriminative latent feature representations and effectively distinguish subtly distinct working memory tasks of different subjects with significantly higher accuracy than conventional decoding models. Informative FNs of the brain decoding models identified as brain activation patterns of working memory tasks were largely consistent with the literature. The method also obtained promising decoding performance on motor and social cognition tasks. Our results suggest that LSTM RNNs in conjunction with FNs could build interpretable, highly accurate brain decoding models.
Collapse
Affiliation(s)
- Hongming Li
- Center for Biomedical Image Computing and Analytics, Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Yong Fan
- Center for Biomedical Image Computing and Analytics, Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, 19104, USA.
| |
Collapse
|
8
|
Branco MP, de Boer LM, Ramsey NF, Vansteensel MJ. Encoding of kinetic and kinematic movement parameters in the sensorimotor cortex: A Brain-Computer Interface perspective. Eur J Neurosci 2019; 50:2755-2772. [PMID: 30633413 PMCID: PMC6625947 DOI: 10.1111/ejn.14342] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2018] [Revised: 11/30/2018] [Accepted: 01/07/2019] [Indexed: 01/23/2023]
Abstract
For severely paralyzed people, Brain-Computer Interfaces (BCIs) can potentially replace lost motor output and provide a brain-based control signal for augmentative and alternative communication devices or neuroprosthetics. Many BCIs focus on neuronal signals acquired from the hand area of the sensorimotor cortex, employing changes in the patterns of neuronal firing or spectral power associated with one or more types of hand movement. Hand and finger movement can be described by two groups of movement features, namely kinematics (spatial and motion aspects) and kinetics (muscles and forces). Despite extensive primate and human research, it is not fully understood how these features are represented in the SMC and how they lead to the appropriate movement. Yet, the available information may provide insight into which features are most suitable for BCI control. To that purpose, the current paper provides an in-depth review on the movement features encoded in the SMC. Even though there is no consensus on how exactly the SMC generates movement, we conclude that some parameters are well represented in the SMC and can be accurately used for BCI control with discrete as well as continuous feedback. However, the vast evidence also suggests that movement should be interpreted as a combination of multiple parameters rather than isolated ones, pleading for further exploration of sensorimotor control models for accurate BCI control.
Collapse
Affiliation(s)
- Mariana P. Branco
- Brain Center Rudolf MagnusDepartment of Neurology and NeurosurgeryUniversity Medical Center UtrechtUtrechtThe Netherlands
| | | | - Nick F. Ramsey
- Brain Center Rudolf MagnusDepartment of Neurology and NeurosurgeryUniversity Medical Center UtrechtUtrechtThe Netherlands
| | - Mariska J. Vansteensel
- Brain Center Rudolf MagnusDepartment of Neurology and NeurosurgeryUniversity Medical Center UtrechtUtrechtThe Netherlands
| |
Collapse
|
9
|
Valente G, Kaas AL, Formisano E, Goebel R. Optimizing fMRI experimental design for MVPA-based BCI control: Combining the strengths of block and event-related designs. Neuroimage 2019; 186:369-381. [DOI: 10.1016/j.neuroimage.2018.10.080] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2018] [Revised: 09/21/2018] [Accepted: 10/30/2018] [Indexed: 11/25/2022] Open
|
10
|
Li H, Fan Y. Brain Decoding from Functional MRI Using Long Short-Term Memory Recurrent Neural Networks. MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION : MICCAI ... INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION 2018; 11072:320-328. [PMID: 30320311 PMCID: PMC6180332 DOI: 10.1007/978-3-030-00931-1_37] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
Abstract
Decoding brain functional states underlying different cognitive processes using multivariate pattern recognition techniques has attracted increasing interests in brain imaging studies. Promising performance has been achieved using brain functional connectivity or brain activation signatures for a variety of brain decoding tasks. However, most of existing studies have built decoding models upon features extracted from imaging data at individual time points or temporal windows with a fixed interval, which might not be optimal across different cognitive processes due to varying temporal durations and dependency of different cognitive processes. In this study, we develop a deep learning based framework for brain decoding by leveraging recent advances in sequence modeling using long short-term memory (LSTM) recurrent neural networks (RNNs). Particularly, functional profiles extracted from task functional imaging data based on their corresponding subject-specific intrinsic functional networks are used as features to build brain decoding models, and LSTM RNNs are adopted to learn decoding mappings between functional profiles and brain states. We evaluate the proposed method using task fMRI data from the HCP dataset, and experimental results have demonstrated that the proposed method could effectively distinguish brain states under different task events and obtain higher accuracy than conventional decoding models.
Collapse
Affiliation(s)
- Hongming Li
- Center for Biomedical Image Computing and Analytics, Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, USA
| | - Yong Fan
- Center for Biomedical Image Computing and Analytics, Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, USA
| |
Collapse
|
11
|
Belyk M, Lee YS, Brown S. How does human motor cortex regulate vocal pitch in singers? ROYAL SOCIETY OPEN SCIENCE 2018; 5:172208. [PMID: 30224990 PMCID: PMC6124115 DOI: 10.1098/rsos.172208] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/14/2017] [Accepted: 07/20/2018] [Indexed: 06/08/2023]
Abstract
Vocal pitch is used as an important communicative device by humans, as found in the melodic dimension of both speech and song. Vocal pitch is determined by the degree of tension in the vocal folds of the larynx, which itself is influenced by complex and nonlinear interactions among the laryngeal muscles. The relationship between these muscles and vocal pitch has been described by a mathematical model in the form of a set of 'control rules'. We searched for the biological implementation of these control rules in the larynx motor cortex of the human brain. We scanned choral singers with functional magnetic resonance imaging as they produced discrete pitches at four different levels across their vocal range. While the locations of the larynx motor activations varied across singers, the activation peaks for the four pitch levels were highly consistent within each individual singer. This result was corroborated using multi-voxel pattern analysis, which demonstrated an absence of patterned activations differentiating any pairing of pitch levels. The complex and nonlinear relationships between the multiple laryngeal muscles that control vocal pitch may obscure the neural encoding of vocal pitch in the brain.
Collapse
Affiliation(s)
- Michel Belyk
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, Ontario, Canada
| | - Yune S. Lee
- Department of Speech and Hearing Sciences and Center for Brain Injury, The Ohio State University, Columbus, OH, USA
| | - Steven Brown
- Department of Psychology, Neuroscience & Behaviour, McMaster University, Hamilton, Ontario, Canada
| |
Collapse
|
12
|
Closed-Loop Control of a Neuroprosthetic Hand by Magnetoencephalographic Signals. PLoS One 2015; 10:e0131547. [PMID: 26134845 PMCID: PMC4489903 DOI: 10.1371/journal.pone.0131547] [Citation(s) in RCA: 24] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2015] [Accepted: 06/03/2015] [Indexed: 11/24/2022] Open
Abstract
Objective A neuroprosthesis using a brain–machine interface (BMI) is a promising therapeutic option for severely paralyzed patients, but the ability to control it may vary among individual patients and needs to be evaluated before any invasive procedure is undertaken. We have developed a neuroprosthetic hand that can be controlled by magnetoencephalographic (MEG) signals to noninvasively evaluate subjects’ ability to control a neuroprosthesis. Method Six nonparalyzed subjects performed grasping or opening movements of their right hand while the slow components of the MEG signals (SMFs) were recorded in an open-loop condition. The SMFs were used to train two decoders to infer the timing and types of movement by support vector machine and Gaussian process regression. The SMFs were also used to calculate estimated slow cortical potentials (eSCPs) to identify the origin of motor information. Finally, using the trained decoders, the subjects controlled a neuroprosthetic hand in a closed-loop condition. Results The SMFs in the open-loop condition revealed movement-related cortical field characteristics and successfully inferred the movement type with an accuracy of 75.0 ± 12.9% (mean ± SD). In particular, the eSCPs in the sensorimotor cortex contralateral to the moved hand varied significantly enough among the movement types to be decoded with an accuracy of 76.5 ± 10.6%, which was significantly higher than the accuracy associated with eSCPs in the ipsilateral sensorimotor cortex (58.1 ± 13.7%; p = 0.0072, paired two-tailed Student’s t-test). Moreover, another decoder using SMFs successfully inferred when the accuracy was the greatest. Combining these two decoders allowed the neuroprosthetic hand to be controlled in a closed-loop condition. Conclusions Use of real-time MEG signals was shown to successfully control the neuroprosthetic hand. The developed system may be useful for evaluating movement-related slow cortical potentials of severely paralyzed patients to predict the efficacy of invasive BMI.
Collapse
|