1
|
Zhang J, Wang Y, Mao Y, Leong C, Yuan Z. Shared Minds, Shared Feedback: tracing the influence of parental feedback on shared neural patterns. Cereb Cortex 2024; 34:bhad489. [PMID: 38163444 DOI: 10.1093/cercor/bhad489] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Revised: 11/27/2023] [Accepted: 11/28/2023] [Indexed: 01/03/2024] Open
Abstract
Parental feedback affects children in multiple ways. However, little is known about how children, family, and feedback types affect parental feedback neural mechanisms. The current study used functional near-infrared spectroscopy-based hyperscanning to observe 47 mother-daughter pairs's (mean age of mothers: 35.95 ± 3.99 yr old; mean age of daughters: 6.97 ± 0.75 yr old) brain synchronization in a jigsaw game under various conditions. Between parental negative feedback and praise conditions, mother-daughter brain in supramarginal gyrus, left dorsolateral prefrontal cortex, right inferior frontal gyrus, and right primary somatic (S1) differed. When criticized, conformity family-communication-patterned families had much worse brain synchronization in S1, left dorsolateral prefrontal cortex, and right Wernicke's region than conversational families. Resilient children had better mother-child supramarginal gyrus synchronicity under negative feedback. This study supports the importance of studying children's neurological development in nurturing environments to assess their psychological development.
Collapse
Affiliation(s)
- Juan Zhang
- Faculty of Education, University of Macau, Avenida da Universidade, Taipa, Macau, China
- Centre for Cognitive and Brain Sciences, University of Macau, Avenida da Universidade, Taipa, Macau, China
| | - Yihui Wang
- Faculty of Education, University of Macau, Avenida da Universidade, Taipa, Macau, China
- Centre for Cognitive and Brain Sciences, University of Macau, Avenida da Universidade, Taipa, Macau, China
| | - Yidi Mao
- Faculty of Education, University of Macau, Avenida da Universidade, Taipa, Macau, China
- Centre for Cognitive and Brain Sciences, University of Macau, Avenida da Universidade, Taipa, Macau, China
| | - Chantat Leong
- Centre for Cognitive and Brain Sciences, University of Macau, Avenida da Universidade, Taipa, Macau, China
- Faculty of Health Sciences, University of Macau, Avenida da Universidade, Taipa, Macau, China
| | - Zhen Yuan
- Centre for Cognitive and Brain Sciences, University of Macau, Avenida da Universidade, Taipa, Macau, China
- Faculty of Health Sciences, University of Macau, Avenida da Universidade, Taipa, Macau, China
| |
Collapse
|
2
|
Canny E, Vansteensel MJ, van der Salm SMA, Müller-Putz GR, Berezutskaya J. Boosting brain-computer interfaces with functional electrical stimulation: potential applications in people with locked-in syndrome. J Neuroeng Rehabil 2023; 20:157. [PMID: 37980536 PMCID: PMC10656959 DOI: 10.1186/s12984-023-01272-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Accepted: 10/23/2023] [Indexed: 11/20/2023] Open
Abstract
Individuals with a locked-in state live with severe whole-body paralysis that limits their ability to communicate with family and loved ones. Recent advances in brain-computer interface (BCI) technology have presented a potential alternative for these people to communicate by detecting neural activity associated with attempted hand or speech movements and translating the decoded intended movements to a control signal for a computer. A technique that could potentially enrich the communication capacity of BCIs is functional electrical stimulation (FES) of paralyzed limbs and face to restore body and facial movements of paralyzed individuals, allowing to add body language and facial expression to communication BCI utterances. Here, we review the current state of the art of existing BCI and FES work in people with paralysis of body and face and propose that a combined BCI-FES approach, which has already proved successful in several applications in stroke and spinal cord injury, can provide a novel promising mode of communication for locked-in individuals.
Collapse
Affiliation(s)
- Evan Canny
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Mariska J Vansteensel
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Sandra M A van der Salm
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Gernot R Müller-Putz
- Institute of Neural Engineering, Laboratory of Brain-Computer Interfaces, Graz University of Technology, Graz, Austria
| | - Julia Berezutskaya
- Department of Neurology and Neurosurgery, Brain Center, University Medical Center Utrecht, Utrecht, The Netherlands.
| |
Collapse
|
3
|
Accou B, Vanthornhout J, Hamme HV, Francart T. Decoding of the speech envelope from EEG using the VLAAI deep neural network. Sci Rep 2023; 13:812. [PMID: 36646740 PMCID: PMC9842721 DOI: 10.1038/s41598-022-27332-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Accepted: 12/30/2022] [Indexed: 01/18/2023] Open
Abstract
To investigate the processing of speech in the brain, commonly simple linear models are used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly-dynamic, complex non-linear system like the brain, and they often require a substantial amount of subject-specific training data. This work introduces a novel speech decoder architecture: the Very Large Augmented Auditory Inference (VLAAI) network. The VLAAI network outperformed state-of-the-art subject-independent models (median Pearson correlation of 0.19, p < 0.001), yielding an increase over the well-established linear model by 52%. Using ablation techniques, we identified the relative importance of each part of the VLAAI network and found that the non-linear components and output context module influenced model performance the most (10% relative performance increase). Subsequently, the VLAAI network was evaluated on a holdout dataset of 26 subjects and a publicly available unseen dataset to test generalization for unseen subjects and stimuli. No significant difference was found between the default test and the holdout subjects, and between the default test set and the public dataset. The VLAAI network also significantly outperformed all baseline models on the public dataset. We evaluated the effect of training set size by training the VLAAI network on data from 1 up to 80 subjects and evaluated on 26 holdout subjects, revealing a relationship following a hyperbolic tangent function between the number of subjects in the training set and the performance on unseen subjects. Finally, the subject-independent VLAAI network was finetuned for 26 holdout subjects to obtain subject-specific VLAAI models. With 5 minutes of data or more, a significant performance improvement was found, up to 34% (from 0.18 to 0.25 median Pearson correlation) with regards to the subject-independent VLAAI network.
Collapse
Affiliation(s)
- Bernd Accou
- ExpORL, Department of Neurosciences, KU Leuven, Leuven, Belgium. .,PSI, Department of Electrical Engineering, KU Leuven, Leuven, Belgium.
| | | | - Hugo Van Hamme
- PSI, Department of Electrical Engineering, KU Leuven, Leuven, Belgium
| | - Tom Francart
- ExpORL, Department of Neurosciences, KU Leuven, Leuven, Belgium.
| |
Collapse
|
4
|
Verwoert M, Ottenhoff MC, Goulis S, Colon AJ, Wagner L, Tousseyn S, van Dijk JP, Kubben PL, Herff C. Dataset of Speech Production in intracranial.Electroencephalography. Sci Data 2022; 9:434. [PMID: 35869138 PMCID: PMC9307753 DOI: 10.1038/s41597-022-01542-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2022] [Accepted: 07/08/2022] [Indexed: 11/28/2022] Open
Abstract
Speech production is an intricate process involving a large number of muscles and cognitive processes. The neural processes underlying speech production are not completely understood. As speech is a uniquely human ability, it can not be investigated in animal models. High-fidelity human data can only be obtained in clinical settings and is therefore not easily available to all researchers. Here, we provide a dataset of 10 participants reading out individual words while we measured intracranial EEG from a total of 1103 electrodes. The data, with its high temporal resolution and coverage of a large variety of cortical and sub-cortical brain regions, can help in understanding the speech production process better. Simultaneously, the data can be used to test speech decoding and synthesis approaches from neural data to develop speech Brain-Computer Interfaces and speech neuroprostheses. Measurement(s) | Brain activity | Technology Type(s) | Stereotactic electroencephalography | Sample Characteristic - Organism | Homo sapiens | Sample Characteristic - Environment | Epilepsy monitoring center | Sample Characteristic - Location | The Netherlands |
Collapse
|
5
|
Loriette C, Amengual JL, Ben Hamed S. Beyond the brain-computer interface: Decoding brain activity as a tool to understand neuronal mechanisms subtending cognition and behavior. Front Neurosci 2022; 16:811736. [PMID: 36161174 PMCID: PMC9492914 DOI: 10.3389/fnins.2022.811736] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2021] [Accepted: 08/23/2022] [Indexed: 11/13/2022] Open
Abstract
One of the major challenges in system neurosciences consists in developing techniques for estimating the cognitive information content in brain activity. This has an enormous potential in different domains spanning from clinical applications, cognitive enhancement to a better understanding of the neural bases of cognition. In this context, the inclusion of machine learning techniques to decode different aspects of human cognition and behavior and its use to develop brain-computer interfaces for applications in neuroprosthetics has supported a genuine revolution in the field. However, while these approaches have been shown quite successful for the study of the motor and sensory functions, success is still far from being reached when it comes to covert cognitive functions such as attention, motivation and decision making. While improvement in this field of BCIs is growing fast, a new research focus has emerged from the development of strategies for decoding neural activity. In this review, we aim at exploring how the advanced in decoding of brain activity is becoming a major neuroscience tool moving forward our understanding of brain functions, providing a robust theoretical framework to test predictions on the relationship between brain activity and cognition and behavior.
Collapse
Affiliation(s)
- Célia Loriette
- Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Université Claude Bernard Lyon 1, Bron, France
| | | | - Suliann Ben Hamed
- Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Université Claude Bernard Lyon 1, Bron, France
| |
Collapse
|
6
|
Sinko L, Regier P, Curtin A, Ayaz H, Rose Childress A, Teitelman AM. Neural correlates of cognitive control in women with a history of sexual violence suggest altered prefrontal cortical activity during cognitive processing. WOMEN'S HEALTH (LONDON, ENGLAND) 2022; 18:17455057221081326. [PMID: 35225075 PMCID: PMC8883288 DOI: 10.1177/17455057221081326] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/12/2021] [Revised: 01/31/2022] [Accepted: 02/01/2022] [Indexed: 06/14/2023]
Abstract
OBJECTIVE Women's experiences of sexual violence can be not only psychologically and physically traumatizing but may also have lasting effects on brain functions, including cognitive control relating to the inhibition and processing of emotion. Thus, the purpose of this pilot study is to explore underlying neural correlates of sexual violence's impact on cognitive control in women. METHODS Thirty women (aged 21-30 years) participants underwent a quantitative survey along with an affect-congruent Go-NoGo task. Prefrontal activity was monitored using functional near-infrared spectroscopy, a portable neuroimaging technology. An analysis of variance tested for main effects of the condition (Go versus NoGo), group (sexual violence versus no prior sexual violence), and potential interactions. RESULTS Fifteen of 30 women reported a history of childhood (n = 5) and/or adult (n = 12) sexual violence. Those with sexual violence histories reported significantly higher depression, anxiety, and posttraumatic stress symptoms, as well as increased impulsivity compared to their peers. Behavioral performance did not differ between the groups; however, functional near-infrared spectroscopy data revealed a significant (group × condition) interaction in Optodes 13 and 16. Women with histories of sexual violence had a significantly lower response during the "NoGo" condition and a heightened response during the "Go" condition, in the right dorsolateral prefrontal cortex. CONCLUSION These results suggest altered prefrontal cortical activity during cognitive processing in women with a history of sexual violence, showing hypoactivity during response inhibition and hyperactivity to the positive stimuli. These findings have strong translational promise for innovative assessment and prevention of untoward effects among women with sexual violence.
Collapse
Affiliation(s)
- Laura Sinko
- Department of Nursing, College of Public Health, Temple University, Philadelphia, PA, USA
| | - Paul Regier
- Department of Psychiatry, Center for Studies of Addiction, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Adrian Curtin
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA, USA
| | - Hasan Ayaz
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA, USA
- School of Nursing, University of Pennsylvania, Philadelphia, PA, USA
- Department of Psychology, College of Arts and Sciences, Drexel University, Philadelphia, PA, USA
- Drexel Solutions Institute, Drexel University, Philadelphia, PA, USA
- Center for Injury Research and Prevention, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Anna Rose Childress
- Department of Psychiatry, Center for Studies of Addiction, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Anne M Teitelman
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA, USA
| |
Collapse
|
7
|
Gaudry KS, Ayaz H, Bedows A, Celnik P, Eagleman D, Grover P, Illes J, Rao RPN, Robinson JT, Thyagarajan K. Projections and the Potential Societal Impact of the Future of Neurotechnologies. Front Neurosci 2021; 15:658930. [PMID: 34867139 PMCID: PMC8634831 DOI: 10.3389/fnins.2021.658930] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2021] [Accepted: 10/04/2021] [Indexed: 12/17/2022] Open
Abstract
Traditionally, recording from and stimulating the brain with high spatial and temporal resolution required invasive means. However, recently, the technical capabilities of less invasive and non-invasive neuro-interfacing technology have been dramatically improving, and laboratories and funders aim to further improve these capabilities. These technologies can facilitate functions such as multi-person communication, mood regulation and memory recall. We consider a potential future where the less invasive technology is in high demand. Will this demand match that the current-day demand for a smartphone? Here, we draw upon existing research to project which particular neuroethics issues may arise in this potential future and what preparatory steps may be taken to address these issues.
Collapse
Affiliation(s)
- Kate S. Gaudry
- Kilpatrick Townsend & Stockton LLP, Washington, DC, United States
| | - Hasan Ayaz
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA, United States
- Department of Psychology, College of Arts and Sciences, Drexel University, Philadelphia, PA, United States
- Drexel Solutions Institute, Drexel University, Philadelphia, PA, United States
- Department of Family and Community Health, University of Pennsylvania, Philadelphia, PA, United States
- Center for Injury Research and Prevention, Children’s Hospital of Philadelphia, Philadelphia, PA, United States
| | | | - Pablo Celnik
- Department of Physical Medicine and Rehabilitation, Johns Hopkins, School of Medicine, Baltimore, MD, United States
| | - David Eagleman
- Department of Psychiatry, Stanford University School of Medicine, Stanford, CA, United States
| | - Pulkit Grover
- Center for the Neural Basis of Cognition, Carnegie Mellon University, Pittsburgh, PA, United States
- Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, United States
| | - Judy Illes
- Department of Medicine, University of British Columbia, Vancouver, BC, Canada
- Neuroethics Canada, University of British Columbia, Vancouver, BC, Canada
| | - Rajesh P. N. Rao
- Center for Neurotechnology, Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, DC, United States
| | - Jacob T. Robinson
- Department of Bioengineering, Rice University, Houston, TX, United States
- Department of Electrical and Computer Engineering, Rice University, Houston, TX, United States
- Applied Physics Program, Rice University, Houston, TX, United States
- Department of Neuroscience, Baylor College of Medicine, Houston, TX, United States
| | | |
Collapse
|
8
|
Tsow F, Kumar A, Hosseini SMH, Bowden A. A low-cost, wearable, do-it-yourself functional near-infrared spectroscopy (DIY-fNIRS) headband. HARDWAREX 2021; 10:e00204. [PMID: 34734152 PMCID: PMC8562714 DOI: 10.1016/j.ohx.2021.e00204] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/26/2020] [Revised: 05/14/2021] [Accepted: 05/16/2021] [Indexed: 05/27/2023]
Abstract
Neuromonitoring in naturalistic environments is of increasing interest for a variety of research fields including psychology, economics, and productivity. Among functional neuromonitoring modalities, functional near-infrared spectroscopy (fNIRS) is well regarded for its potential for miniaturization, good spatial and temporal resolutions, and resilience to motion artifacts. Historically, the large size and high cost of fNIRS systems have precluded widespread adoption of the technology. In this article, we describe the first open source, fully integrated wireless fNIRS headband system with a single LED-pair source and four detectors. With ease of operation and comfort in mind, the system is encased in a soft, lightweight cloth and silicone enclosure. Accompanying computer and smartphone data collection software have also been provided, and the hardware has been validated using classic fNIRS tasks. This wear-and-go design can easily be scaled to accommodate a larger number of fNIRS channels and opens the door to easily collecting fNIRS data during routine activities in naturalistic conditions.
Collapse
Affiliation(s)
- Francis Tsow
- Biophotonics Center, Vanderbilt University, Nashville, TN 37232, United States
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN 37232, United States
| | - Anupam Kumar
- Biophotonics Center, Vanderbilt University, Nashville, TN 37232, United States
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN 37232, United States
| | - SM Hadi Hosseini
- Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA 94305, United States
| | - Audrey Bowden
- Biophotonics Center, Vanderbilt University, Nashville, TN 37232, United States
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN 37232, United States
- Department of Electrical Engineering, Vanderbilt University, Nashville, TN 37232, United States
| |
Collapse
|
9
|
Barreto C, Bruneri GDA, Brockington G, Ayaz H, Sato JR. A New Statistical Approach for fNIRS Hyperscanning to Predict Brain Activity of Preschoolers' Using Teacher's. Front Hum Neurosci 2021; 15:622146. [PMID: 34025373 PMCID: PMC8137814 DOI: 10.3389/fnhum.2021.622146] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/27/2020] [Accepted: 03/17/2021] [Indexed: 11/18/2022] Open
Abstract
Hyperscanning studies using functional Near-Infrared Spectroscopy (fNIRS) have been performed to understand the neural mechanisms underlying human-human interactions. In this study, we propose a novel methodological approach that is developed for fNIRS multi-brain analysis. Our method uses support vector regression (SVR) to predict one brain activity time series using another as the predictor. We applied the proposed methodology to explore the teacher-student interaction, which plays a critical role in the formal learning process. In an illustrative application, we collected fNIRS data of the teacher and preschoolers’ dyads performing an interaction task. The teacher explained to the child how to add two numbers in the context of a game. The Prefrontal cortex and temporal-parietal junction of both teacher and student were recorded. A multivariate regression model was built for each channel in each dyad, with the student’s signal as the response variable and the teacher’s ones as the predictors. We compared the predictions of SVR with the conventional ordinary least square (OLS) predictor. The results predicted by the SVR model were statistically significantly correlated with the actual test data at least one channel-pair for all dyads. Overall, 29/90 channel-pairs across the five dyads (18 channels 5 dyads = 90 channel-pairs) presented significant signal predictions withthe SVR approach. The conventional OLS resulted in only 4 out of 90 valid predictions. These results demonstrated that the SVR could be used to perform channel-wise predictions across individuals, and the teachers’ cortical activity can be used to predict the student brain hemodynamic response.
Collapse
Affiliation(s)
- Candida Barreto
- Center of Mathematics, Computing and Cognition, Federal University of ABC, Sao Bernardo do Campo, Brazil
| | | | - Guilherme Brockington
- Center for Natural and Human Sciences, Universidade Federal do ABC, Santo André, Brazil
| | - Hasan Ayaz
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA, United States.,Department of Psychology, College of Arts and Sciences, Drexel University, Philadelphia, PA, United States.,Drexel Solutions Institute, Drexel University, Philadelphia, PA, United States.,Department of Family and Community Health, University of Pennsylvania, Philadelphia, PA, United States.,Center for Injury Research and Prevention, Children's Hospital of Philadelphia, Philadelphia, PA, United States
| | - Joao Ricardo Sato
- Center of Mathematics, Computing and Cognition, Federal University of ABC, Sao Bernardo do Campo, Brazil.,Interdisciplinary Unit for Applied Neuroscience, Federal University of ABC, Sao Bernardo do Campo, Brazil
| |
Collapse
|
10
|
Oku AYA, Sato JR. Predicting Student Performance Using Machine Learning in fNIRS Data. Front Hum Neurosci 2021; 15:622224. [PMID: 33613215 PMCID: PMC7892769 DOI: 10.3389/fnhum.2021.622224] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2020] [Accepted: 01/08/2021] [Indexed: 11/13/2022] Open
Abstract
Increasing student involvement in classes has always been a challenge for teachers and school managers. In online learning, some interactivity mechanisms like quizzes are increasingly used to engage students during classes and tasks. However, there is a high demand for tools that evaluate the efficiency of these mechanisms. In order to distinguish between high and low levels of engagement in tasks, it is possible to monitor brain activity through functional near-infrared spectroscopy (fNIRS). The main advantages of this technique are portability, low cost, and a comfortable way for students to concentrate and perform their tasks. This setup provides more natural conditions for the experiments if compared to the other acquisition tools. In this study, we investigated levels of task involvement through the identification of correct and wrong answers of typical quizzes used in virtual environments. We collected data from the prefrontal cortex region (PFC) of 18 students while watching a video lecture. This data was modeled with supervised learning algorithms. We used random forests and penalized logistic regression to classify correct answers as a function of oxyhemoglobin and deoxyhemoglobin concentration. These models identify which regions best predict student performance. The random forest and penalized logistic regression (GLMNET with LASSO) obtained, respectively, 0.67 and 0.65 area of the ROC curve. Both models indicate that channels F4-F6 and AF3-AFz are the most relevant for the prediction. The statistical significance of these models was confirmed through cross-validation (leave-one-subject-out) and a permutation test. This methodology can be useful to better understand the teaching and learning processes in a video lecture and also provide improvements in the methodologies used in order to better adapt the presentation content.
Collapse
Affiliation(s)
- Amanda Yumi Ambriola Oku
- Center of Mathematics, Computing and Cognition, Universidade Federal Do ABC, São Bernardo Do Campo, Brazil
| | - João Ricardo Sato
- Center of Mathematics, Computing and Cognition, Universidade Federal Do ABC, São Bernardo Do Campo, Brazil
| |
Collapse
|
11
|
Zhuang X, Yang Z, Cordes D. A technical review of canonical correlation analysis for neuroscience applications. Hum Brain Mapp 2020; 41:3807-3833. [PMID: 32592530 PMCID: PMC7416047 DOI: 10.1002/hbm.25090] [Citation(s) in RCA: 70] [Impact Index Per Article: 17.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2020] [Accepted: 05/23/2020] [Indexed: 12/11/2022] Open
Abstract
Collecting comprehensive data sets of the same subject has become a standard in neuroscience research and uncovering multivariate relationships among collected data sets have gained significant attentions in recent years. Canonical correlation analysis (CCA) is one of the powerful multivariate tools to jointly investigate relationships among multiple data sets, which can uncover disease or environmental effects in various modalities simultaneously and characterize changes during development, aging, and disease progressions comprehensively. In the past 10 years, despite an increasing number of studies have utilized CCA in multivariate analysis, simple conventional CCA dominates these applications. Multiple CCA-variant techniques have been proposed to improve the model performance; however, the complicated multivariate formulations and not well-known capabilities have delayed their wide applications. Therefore, in this study, a comprehensive review of CCA and its variant techniques is provided. Detailed technical formulation with analytical and numerical solutions, current applications in neuroscience research, and advantages and limitations of each CCA-related technique are discussed. Finally, a general guideline in how to select the most appropriate CCA-related technique based on the properties of available data sets and particularly targeted neuroscience questions is provided.
Collapse
Affiliation(s)
- Xiaowei Zhuang
- Cleveland Clinic Lou Ruvo Center for Brain HealthLas VegasNevadaUSA
| | - Zhengshi Yang
- Cleveland Clinic Lou Ruvo Center for Brain HealthLas VegasNevadaUSA
| | - Dietmar Cordes
- Cleveland Clinic Lou Ruvo Center for Brain HealthLas VegasNevadaUSA
- University of ColoradoBoulderColoradoUSA
- Department of Brain HealthUniversity of NevadaLas VegasNevadaUSA
| |
Collapse
|
12
|
Asgher U, Khalil K, Khan MJ, Ahmad R, Butt SI, Ayaz Y, Naseer N, Nazir S. Enhanced Accuracy for Multiclass Mental Workload Detection Using Long Short-Term Memory for Brain-Computer Interface. Front Neurosci 2020; 14:584. [PMID: 32655353 PMCID: PMC7324788 DOI: 10.3389/fnins.2020.00584] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2019] [Accepted: 05/12/2020] [Indexed: 11/30/2022] Open
Abstract
Cognitive workload is one of the widely invoked human factors in the areas of human-machine interaction (HMI) and neuroergonomics. The precise assessment of cognitive and mental workload (MWL) is vital and requires accurate neuroimaging to monitor and evaluate the cognitive states of the brain. In this study, we have decoded four classes of MWL using long short-term memory (LSTM) with 89.31% average accuracy for brain-computer interface (BCI). The brain activity signals are acquired using functional near-infrared spectroscopy (fNIRS) from the prefrontal cortex (PFC) region of the brain. We performed a supervised MWL experimentation with four varying MWL levels on 15 participants (both male and female) and 10 trials of each MWL per participant. Real-time four-level MWL states are assessed using fNIRS system, and initial classification is performed using three strong machine learning (ML) techniques, support vector machine (SVM), k-nearest neighbor (k-NN), and artificial neural network (ANN) with obtained average accuracies of 54.33, 54.31, and 69.36%, respectively. In this study, novel deep learning (DL) frameworks are proposed, which utilizes convolutional neural network (CNN) and LSTM with 87.45 and 89.31% average accuracies, respectively, to solve high-dimensional four-level cognitive states classification problem. Statistical analysis, t-test, and one-way F-test (ANOVA) are also performed on accuracies obtained through ML and DL algorithms. Results show that the proposed DL (LSTM and CNN) algorithms significantly improve classification performance as compared with ML (SVM, ANN, and k-NN) algorithms.
Collapse
Affiliation(s)
- Umer Asgher
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Khurram Khalil
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Muhammad Jawad Khan
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Riaz Ahmad
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
- Directorate of Quality Assurance and International Collaboration, National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Shahid Ikramullah Butt
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
| | - Yasar Ayaz
- School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, Pakistan
- National Center of Artificial Intelligence (NCAI) – NUST, Islamabad, Pakistan
| | - Noman Naseer
- Department of Mechatronics Engineering, Air University, Islamabad, Pakistan
| | - Salman Nazir
- Training and Assessment Research Group, Department of Maritime Operations, University of South-Eastern Norway, Kongsberg, Norway
| |
Collapse
|
13
|
Brain–machine interfaces using functional near-infrared spectroscopy: a review. ARTIFICIAL LIFE AND ROBOTICS 2020. [DOI: 10.1007/s10015-020-00592-9] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
|
14
|
Keshmiri S, Sumioka H, Yamazaki R, Shiomi M, Ishiguro H. Information Content of Prefrontal Cortex Activity Quantifies the Difficulty of Narrated Stories. Sci Rep 2019; 9:17959. [PMID: 31784577 PMCID: PMC6884437 DOI: 10.1038/s41598-019-54280-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2019] [Accepted: 11/11/2019] [Indexed: 12/14/2022] Open
Abstract
The ability to realize the individuals' impressions during the verbal communication allows social robots to significantly facilitate their social interactions in such areas as child education and elderly care. However, such impressions are highly subjective and internalized and therefore cannot be easily comprehended through behavioural observations. Although brain-machine interface suggests the utility of the brain information in human-robot interaction, previous studies did not consider its potential for estimating the internal impressions during verbal communication. In this article, we introduce a novel approach to estimation of the individuals' perceived difficulty of stories using the quantified information content of their prefrontal cortex activity. We demonstrate the robustness of our approach by showing its comparable performance in face-to-face, humanoid, speaker, and video-chat settings. Our results contribute to the field of socially assistive robotics by taking a step toward enabling robots determine their human companions' perceived difficulty of conversations, thereby enabling these media to sustain their communication with humans by adapting to individuals' pace and interest in response to conversational nuances and complexity.
Collapse
Affiliation(s)
- Soheil Keshmiri
- Advanced Telecommunications Research Institute International (ATR), Kyoto, Japan.
| | - Hidenobu Sumioka
- Advanced Telecommunications Research Institute International (ATR), Kyoto, Japan
| | - Ryuji Yamazaki
- Symbiotic Intelligent Systems Research Center, Institute for Open and Transdisciplinary Research Initiatives, Osaka University, Osaka, Japan
| | - Masahiro Shiomi
- Advanced Telecommunications Research Institute International (ATR), Kyoto, Japan
| | - Hiroshi Ishiguro
- Advanced Telecommunications Research Institute International (ATR), Kyoto, Japan
- Graduate School of Engineering Science, Osaka University, Osaka, Japan
| |
Collapse
|