1
|
Ivanov N, Lio A, Chau T. Towards user-centric BCI design: Markov chain-based user assessment for mental imagery EEG-BCIs. J Neural Eng 2023; 20:066037. [PMID: 38128128 DOI: 10.1088/1741-2552/ad17f2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2023] [Accepted: 12/21/2023] [Indexed: 12/23/2023]
Abstract
Objective.While electroencephalography (EEG)-based brain-computer interfaces (BCIs) have many potential clinical applications, their use is impeded by poor performance for many users. To improve BCI performance, either via enhanced signal processing or user training, it is critical to understand and describe each user's ability to perform mental control tasks and produce discernible EEG patterns. While classification accuracy has predominantly been used to assess user performance, limitations and criticisms of this approach have emerged, thus prompting the need to develop novel user assessment approaches with greater descriptive capability. Here, we propose a combination of unsupervised clustering and Markov chain models to assess and describe user skill.Approach.Using unsupervisedK-means clustering, we segmented the EEG signal space into regions representing pattern states that users could produce. A user's movement through these pattern states while performing different tasks was modeled using Markov chains. Finally, using the steady-state distributions and entropy rates of the Markov chains, we proposed two metricstaskDistinctandrelativeTaskInconsistencyto assess, respectively, a user's ability to (i) produce distinct task-specific patterns for each mental task and (ii) maintain consistent patterns during individual tasks.Main results.Analysis of data from 14 adolescents using a three-class BCI revealed significant correlations between thetaskDistinctandrelativeTaskInconsistencymetrics and classification F1 score. Moreover, analysis of the pattern states and Markov chain models yielded descriptive information regarding user performance not immediately apparent from classification accuracy.Significance.Our proposed user assessment method can be used in concert with classifier-based analysis to further understand the extent to which users produce task-specific, time-evolving EEG patterns. In turn, this information could be used to enhance user training or classifier design.
Collapse
Affiliation(s)
- Nicolas Ivanov
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| | - Aaron Lio
- Division of Engineering Science, University of Toronto, Toronto, ON, Canada
| | - Tom Chau
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
2
|
Floreani ED, Orlandi S, Chau T. A pediatric near-infrared spectroscopy brain-computer interface based on the detection of emotional valence. Front Hum Neurosci 2022; 16:938708. [PMID: 36211121 PMCID: PMC9540519 DOI: 10.3389/fnhum.2022.938708] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2022] [Accepted: 09/05/2022] [Indexed: 11/27/2022] Open
Abstract
Brain-computer interfaces (BCIs) are being investigated as an access pathway to communication for individuals with physical disabilities, as the technology obviates the need for voluntary motor control. However, to date, minimal research has investigated the use of BCIs for children. Traditional BCI communication paradigms may be suboptimal given that children with physical disabilities may face delays in cognitive development and acquisition of literacy skills. Instead, in this study we explored emotional state as an alternative access pathway to communication. We developed a pediatric BCI to identify positive and negative emotional states from changes in hemodynamic activity of the prefrontal cortex (PFC). To train and test the BCI, 10 neurotypical children aged 8–14 underwent a series of emotion-induction trials over four experimental sessions (one offline, three online) while their brain activity was measured with functional near-infrared spectroscopy (fNIRS). Visual neurofeedback was used to assist participants in regulating their emotional states and modulating their hemodynamic activity in response to the affective stimuli. Child-specific linear discriminant classifiers were trained on cumulatively available data from previous sessions and adaptively updated throughout each session. Average online valence classification exceeded chance across participants by the last two online sessions (with 7 and 8 of the 10 participants performing better than chance, respectively, in Sessions 3 and 4). There was a small significant positive correlation with online BCI performance and age, suggesting older participants were more successful at regulating their emotional state and/or brain activity. Variability was seen across participants in regards to BCI performance, hemodynamic response, and discriminatory features and channels. Retrospective offline analyses yielded accuracies comparable to those reported in adult affective BCI studies using fNIRS. Affective fNIRS-BCIs appear to be feasible for school-aged children, but to further gauge the practical potential of this type of BCI, replication with more training sessions, larger sample sizes, and end-users with disabilities is necessary.
Collapse
Affiliation(s)
- Erica D. Floreani
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
- *Correspondence: Erica D. Floreani
| | - Silvia Orlandi
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Department of Biomedical Engineering, University of Bologna, Bologna, Italy
| | - Tom Chau
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Toronto, ON, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
3
|
Khalil K, Asgher U, Ayaz Y. Novel fNIRS study on homogeneous symmetric feature-based transfer learning for brain-computer interface. Sci Rep 2022; 12:3198. [PMID: 35210460 PMCID: PMC8873341 DOI: 10.1038/s41598-022-06805-4] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2021] [Accepted: 01/04/2022] [Indexed: 01/23/2023] Open
Abstract
The brain-computer interface (BCI) provides an alternate means of communication between the brain and external devices by recognizing the brain activities and translating them into external commands. The functional Near-Infrared Spectroscopy (fNIRS) is becoming popular as a non-invasive modality for brain activity detection. The recent trends show that deep learning has significantly enhanced the performance of the BCI systems. But the inherent bottleneck for deep learning (in the domain of BCI) is the requirement of the vast amount of training data, lengthy recalibrating time, and expensive computational resources for training deep networks. Building a high-quality, large-scale annotated dataset for deep learning-based BCI systems is exceptionally tedious, complex, and expensive. This study investigates the novel application of transfer learning for fNIRS-based BCI to solve three objective functions (concerns), i.e., the problem of insufficient training data, reduced training time, and increased accuracy. We applied symmetric homogeneous feature-based transfer learning on convolutional neural network (CNN) designed explicitly for fNIRS data collected from twenty-six (26) participants performing the n-back task. The results suggested that the proposed method achieves the maximum saturated accuracy sooner and outperformed the traditional CNN model on averaged accuracy by 25.58% in the exact duration of training time, reducing the training time, recalibrating time, and computational resources.
Collapse
Affiliation(s)
- Khurram Khalil
- National Center of Artificial Intelligence (NCAI), School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan
| | - Umer Asgher
- National Center of Artificial Intelligence (NCAI), School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan.,Department of Mechatronics Engineering, College of Electrical and Mechanical Engineering, National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan
| | - Yasar Ayaz
- National Center of Artificial Intelligence (NCAI), School of Mechanical and Manufacturing Engineering (SMME), National University of Sciences and Technology (NUST), Islamabad, 44000, Pakistan.
| |
Collapse
|
4
|
Stojic F, Chau T. Nonspecific Visuospatial Imagery as a Novel Mental Task for Online EEG-Based BCI Control. Int J Neural Syst 2020; 30:2050026. [PMID: 32498642 DOI: 10.1142/s0129065720500264] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
Brain-computer interfaces (BCIs) can provide a means of communication to individuals with severe motor disorders, such as those presenting as locked-in. Many BCI paradigms rely on motor neural pathways, which are often impaired in these individuals. However, recent findings suggest that visuospatial function may remain intact. This study aimed to determine whether visuospatial imagery, a previously unexplored task, could be used to signify intent in an online electroencephalography (EEG)-based BCI. Eighteen typically developed participants imagined checkerboard arrow stimuli in four quadrants of the visual field in 5-s trials, while signals were collected using 16 dry electrodes over the visual cortex. In online blocks, participants received graded visual feedback based on their performance. An initial BCI pipeline (visuospatial imagery classifier I) attained a mean accuracy of [Formula: see text]% classifying rest against visuospatial imagery in online trials. This BCI pipeline was further improved using restriction to alpha band features (visuospatial imagery classifier II), resulting in a mean pseudo-online accuracy of [Formula: see text]%. Accuracies exceeded the threshold for practical BCIs in 12 participants. This study supports the use of visuospatial imagery as a real-time, binary EEG-BCI control paradigm.
Collapse
Affiliation(s)
- Filip Stojic
- Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, Canada.,Institute of Biomaterials and Biomedical Engineering, University of Toronto, 27 King's College Circle, Toronto, Ontario, Canada M5S 1A1, Canada.,Terrance Donnelly Centre for Cellular and Biomolecular Research, 160 College St, Toronto, Ontario, Canada M5S 3E1, Canada
| | - Tom Chau
- Paediatric Rehabilitation Intelligent Systems, Multidisciplinary (PRISM) Laboratory, 150 Kilgour Rd, East York, Ontario, Canada M4G 1R8, Canada
| |
Collapse
|
5
|
Kohl SH, Mehler DMA, Lührs M, Thibault RT, Konrad K, Sorger B. The Potential of Functional Near-Infrared Spectroscopy-Based Neurofeedback-A Systematic Review and Recommendations for Best Practice. Front Neurosci 2020; 14:594. [PMID: 32848528 PMCID: PMC7396619 DOI: 10.3389/fnins.2020.00594] [Citation(s) in RCA: 53] [Impact Index Per Article: 13.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2020] [Accepted: 05/14/2020] [Indexed: 01/04/2023] Open
Abstract
Background: The effects of electroencephalography (EEG) and functional magnetic resonance imaging (fMRI)-neurofeedback on brain activation and behaviors have been studied extensively in the past. More recently, researchers have begun to investigate the effects of functional near-infrared spectroscopy-based neurofeedback (fNIRS-neurofeedback). FNIRS is a functional neuroimaging technique based on brain hemodynamics, which is easy to use, portable, inexpensive, and has reduced sensitivity to movement artifacts. Method: We provide the first systematic review and database of fNIRS-neurofeedback studies, synthesizing findings from 22 peer-reviewed studies (including a total of N = 441 participants; 337 healthy, 104 patients). We (1) give a comprehensive overview of how fNIRS-neurofeedback training protocols were implemented, (2) review the online signal-processing methods used, (3) evaluate the quality of studies using pre-set methodological and reporting quality criteria and also present statistical sensitivity/power analyses, (4) investigate the effectiveness of fNIRS-neurofeedback in modulating brain activation, and (5) review its effectiveness in changing behavior in healthy and pathological populations. Results and discussion: (1–2) Published studies are heterogeneous (e.g., neurofeedback targets, investigated populations, applied training protocols, and methods). (3) Large randomized controlled trials are still lacking. In view of the novelty of the field, the quality of the published studies is moderate. We identified room for improvement in reporting important information and statistical power to detect realistic effects. (4) Several studies show that people can regulate hemodynamic signals from cortical brain regions with fNIRS-neurofeedback and (5) these studies indicate the feasibility of modulating motor control and prefrontal brain functioning in healthy participants and ameliorating symptoms in clinical populations (stroke, ADHD, autism, and social anxiety). However, valid conclusions about specificity or potential clinical utility are premature. Conclusion: Due to the advantages of practicability and relatively low cost, fNIRS-neurofeedback might provide a suitable and powerful alternative to EEG and fMRI neurofeedback and has great potential for clinical translation of neurofeedback. Together with more rigorous research and reporting practices, further methodological improvements may lead to a more solid understanding of fNIRS-neurofeedback. Future research will benefit from exploiting the advantages of fNIRS, which offers unique opportunities for neurofeedback research.
Collapse
Affiliation(s)
- Simon H Kohl
- JARA-Institute Molecular Neuroscience and Neuroimaging (INM-11), Jülich Research Centre, Jülich, Germany.,Child Neuropsychology Section, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Medical Faculty, RWTH Aachen University, Aachen, Germany
| | - David M A Mehler
- Department of Psychiatry, University of Münster, Münster, Germany
| | - Michael Lührs
- Brain Innovation B.V., Research Department, Maastricht, Netherlands.,Faculty of Psychology and Neuroscience, Department of Cognitive Neuroscience, Maastricht University, Maastricht, Netherlands
| | - Robert T Thibault
- School of Psychological Science, University of Bristol, Bristol, United Kingdom.,MRC Integrative Epidemiology Unit, University of Bristol, Bristol, United Kingdom
| | - Kerstin Konrad
- JARA-Institute Molecular Neuroscience and Neuroimaging (INM-11), Jülich Research Centre, Jülich, Germany.,Child Neuropsychology Section, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Medical Faculty, RWTH Aachen University, Aachen, Germany
| | - Bettina Sorger
- Faculty of Psychology and Neuroscience, Department of Cognitive Neuroscience, Maastricht University, Maastricht, Netherlands
| |
Collapse
|
6
|
Milekovic T, Bacher D, Sarma AA, Simeral JD, Saab J, Pandarinath C, Yvert B, Sorice BL, Blabe C, Oakley EM, Tringale KR, Eskandar E, Cash SS, Shenoy KV, Henderson JM, Hochberg LR, Donoghue JP. Volitional control of single-electrode high gamma local field potentials by people with paralysis. J Neurophysiol 2019; 121:1428-1450. [PMID: 30785814 DOI: 10.1152/jn.00131.2018] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022] Open
Abstract
Intracortical brain-computer interfaces (BCIs) can enable individuals to control effectors, such as a computer cursor, by directly decoding the user's movement intentions from action potentials and local field potentials (LFPs) recorded within the motor cortex. However, the accuracy and complexity of effector control achieved with such "biomimetic" BCIs will depend on the degree to which the intended movements used to elicit control modulate the neural activity. In particular, channels that do not record distinguishable action potentials and only record LFP modulations may be of limited use for BCI control. In contrast, a biofeedback approach may surpass these limitations by letting the participants generate new control signals and learn strategies that improve the volitional control of signals used for effector control. Here, we show that, by using a biofeedback paradigm, three individuals with tetraplegia achieved volitional control of gamma LFPs (40-400 Hz) recorded by a single microelectrode implanted in the precentral gyrus. Control was improved over a pair of consecutive sessions up to 3 days apart. In all but one session, the channel used to achieve control lacked distinguishable action potentials. Our results indicate that biofeedback LFP-based BCIs may potentially contribute to the neural modulation necessary to obtain reliable and useful control of effectors. NEW & NOTEWORTHY Our study demonstrates that people with tetraplegia can volitionally control individual high-gamma local-field potential (LFP) channels recorded from the motor cortex, and that this control can be improved using biofeedback. Motor cortical LFP signals are thought to be both informative and stable intracortical signals and, thus, of importance for future brain-computer interfaces.
Collapse
Affiliation(s)
- Tomislav Milekovic
- Department of Neuroscience, Brown University , Providence, Rhode Island.,Carney Institute for Brain Science, Brown University , Providence, Rhode Island.,Department of Fundamental Neuroscience, Faculty of Medicine, University of Geneva , Geneva , Switzerland
| | - Daniel Bacher
- Carney Institute for Brain Science, Brown University , Providence, Rhode Island.,School of Engineering, Brown University , Providence, Rhode Island
| | - Anish A Sarma
- Carney Institute for Brain Science, Brown University , Providence, Rhode Island.,School of Engineering, Brown University , Providence, Rhode Island.,Center for Neurorestoration and Neurotechnology, Rehabilitation Research & Development Service, Department of Veterans Affairs , Providence, Rhode Island
| | - John D Simeral
- Carney Institute for Brain Science, Brown University , Providence, Rhode Island.,School of Engineering, Brown University , Providence, Rhode Island.,Center for Neurorestoration and Neurotechnology, Rehabilitation Research & Development Service, Department of Veterans Affairs , Providence, Rhode Island
| | - Jad Saab
- Carney Institute for Brain Science, Brown University , Providence, Rhode Island.,School of Engineering, Brown University , Providence, Rhode Island
| | - Chethan Pandarinath
- Department of Neurosurgery, Stanford University , Stanford, California.,Department of Electrical Engineering, Stanford University , Stanford, California.,Stanford Neurosciences Institute, Stanford University , Stanford, California
| | - Blaise Yvert
- Department of Neuroscience, Brown University , Providence, Rhode Island.,Carney Institute for Brain Science, Brown University , Providence, Rhode Island.,Inserm, University of Grenoble, Clinatec-Lab U1205, Grenoble , France
| | - Brittany L Sorice
- Department of Neurology, Massachusetts General Hospital , Boston, Massachusetts
| | - Christine Blabe
- Department of Neurosurgery, Stanford University , Stanford, California
| | - Erin M Oakley
- Department of Neurology, Massachusetts General Hospital , Boston, Massachusetts
| | - Kathryn R Tringale
- Department of Neurology, Massachusetts General Hospital , Boston, Massachusetts
| | - Emad Eskandar
- Department of Neurosurgery, Massachusetts General Hospital , Boston, Massachusetts.,Harvard Medical School , Boston, Massachusetts
| | - Sydney S Cash
- Department of Neurology, Massachusetts General Hospital , Boston, Massachusetts.,Harvard Medical School , Boston, Massachusetts
| | - Krishna V Shenoy
- Department of Electrical Engineering, Stanford University , Stanford, California.,Stanford Neurosciences Institute, Stanford University , Stanford, California.,Neurosciences Program, Stanford University , Stanford, California.,Department of Neurobiology, Stanford University , Stanford, California.,Department of Bioengineering, Stanford University , Stanford, California
| | - Jaimie M Henderson
- Department of Neurosurgery, Stanford University , Stanford, California.,Stanford Neurosciences Institute, Stanford University , Stanford, California.,Department of Neurology and Neurological Sciences, Stanford University , Stanford, California
| | - Leigh R Hochberg
- Carney Institute for Brain Science, Brown University , Providence, Rhode Island.,School of Engineering, Brown University , Providence, Rhode Island.,Center for Neurorestoration and Neurotechnology, Rehabilitation Research & Development Service, Department of Veterans Affairs , Providence, Rhode Island.,Department of Neurology, Massachusetts General Hospital , Boston, Massachusetts.,Harvard Medical School , Boston, Massachusetts
| | - John P Donoghue
- Department of Neuroscience, Brown University , Providence, Rhode Island.,Carney Institute for Brain Science, Brown University , Providence, Rhode Island.,Center for Neurorestoration and Neurotechnology, Rehabilitation Research & Development Service, Department of Veterans Affairs , Providence, Rhode Island
| |
Collapse
|
7
|
Khan MJ, Ghafoor U, Hong KS. Early Detection of Hemodynamic Responses Using EEG: A Hybrid EEG-fNIRS Study. Front Hum Neurosci 2018; 12:479. [PMID: 30555313 PMCID: PMC6281984 DOI: 10.3389/fnhum.2018.00479] [Citation(s) in RCA: 46] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2018] [Accepted: 11/15/2018] [Indexed: 01/06/2023] Open
Abstract
Enhanced classification accuracy and a sufficient number of commands are highly demanding in brain computer interfaces (BCIs). For a successful BCI, early detection of brain commands in time is essential. In this paper, we propose a novel classifier using a modified vector phase diagram and the power of electroencephalography (EEG) signal for early prediction of hemodynamic responses. EEG and functional near-infrared spectroscopy (fNIRS) signals for a motor task (thumb tapping) were obtained concurrently. Upon the resting state threshold circle in the vector phase diagram that uses the maximum values of oxy- and deoxy-hemoglobin (ΔHbO and ΔHbR) during the resting state, we introduce a secondary (inner) threshold circle using the ΔHbO and ΔHbR magnitudes during the time window of 1 s where an EEG activity is noticeable. If the trajectory of ΔHbO and ΔHbR touches the resting state threshold circle after passing through the inner circle, this indicates that ΔHbO was increasing and ΔHbR was decreasing (i.e., the start of a hemodynamic response). It takes about 0.5 s for an fNIRS signal to cross the resting state threshold circle after crossing the EEG-based circle. Thus, an fNIRS-based BCI command can be generated in 1.5 s. We achieved an improved accuracy of 86.0% using the proposed method in comparison with the 63.8% accuracy obtained using linear discriminant analysis in a window of 0~1.5 s. Moreover, the active brain locations (identified using the proposed scheme) were spatially specific when a t-map was made after 10 s of stimulation. These results demonstrate the possibility of enhancing the classification accuracy for a brain-computer interface with a time window of 1.5 s using the proposed method.
Collapse
Affiliation(s)
- M Jawad Khan
- School of Mechanical Engineering, Pusan National University, Busan, South Korea.,School of Mechanical and Manufacturing Engineering, National University of Science and Technology, Islamabad, Pakistan
| | - Usman Ghafoor
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
| | - Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Busan, South Korea.,Department of Cogno-Mechatronics Engineering, Pusan National University, Busan, South Korea
| |
Collapse
|
8
|
Hong KS, Khan MJ, Hong MJ. Feature Extraction and Classification Methods for Hybrid fNIRS-EEG Brain-Computer Interfaces. Front Hum Neurosci 2018; 12:246. [PMID: 30002623 PMCID: PMC6032997 DOI: 10.3389/fnhum.2018.00246] [Citation(s) in RCA: 105] [Impact Index Per Article: 17.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2018] [Accepted: 05/29/2018] [Indexed: 11/13/2022] Open
Abstract
In this study, a brain-computer interface (BCI) framework for hybrid functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) for locked-in syndrome (LIS) patients is investigated. Brain tasks, channel selection methods, and feature extraction and classification algorithms available in the literature are reviewed. First, we categorize various types of patients with cognitive and motor impairments to assess the suitability of BCI for each of them. The prefrontal cortex is identified as a suitable brain region for imaging. Second, the brain activity that contributes to the generation of hemodynamic signals is reviewed. Mental arithmetic and word formation tasks are found to be suitable for use with LIS patients. Third, since a specific targeted brain region is needed for BCI, methods for determining the region of interest are reviewed. The combination of a bundled-optode configuration and threshold-integrated vector phase analysis turns out to be a promising solution. Fourth, the usable fNIRS features and EEG features are reviewed. For hybrid BCI, a combination of the signal peak and mean fNIRS signals and the highest band powers of EEG signals is promising. For classification, linear discriminant analysis has been most widely used. However, further research on vector phase analysis as a classifier for multiple commands is desirable. Overall, proper brain region identification and proper selection of features will improve classification accuracy. In conclusion, five future research issues are identified, and a new BCI scheme, including brain therapy for LIS patients and using the framework of hybrid fNIRS-EEG BCI, is provided.
Collapse
Affiliation(s)
- Keum-Shik Hong
- Department of Cogno-Mechatronics Engineering, Pusan National University, Busan, South Korea.,School of Mechanical Engineering, Pusan National University, Busan, South Korea
| | - M Jawad Khan
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
| | - Melissa J Hong
- Early Learning, FIRST 5 Santa Clara County, San Jose, CA, United States
| |
Collapse
|
9
|
Liu YH, Lin LF, Chou CW, Chang Y, Hsiao YT, Hsu WC. Analysis of Electroencephalography Event-Related Desynchronisation and Synchronisation Induced by Lower-Limb Stepping Motor Imagery. J Med Biol Eng 2018. [DOI: 10.1007/s40846-018-0379-9] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
10
|
Schudlo LC, Chau T. Development of a Ternary Near-Infrared Spectroscopy Brain-Computer Interface: Online Classification of Verbal Fluency Task, Stroop Task and Rest. Int J Neural Syst 2017; 28:1750052. [PMID: 29281922 DOI: 10.1142/s0129065717500526] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
The majority of proposed NIRS-BCIs has considered binary classification. Studies considering high-order classification problems have yielded average accuracies that are less than favorable for practical communication. Consequently, there is a paucity of evidence supporting online classification of more than two mental states using NIRS. We developed an online ternary NIRS-BCI that supports the verbal fluency task (VFT), Stroop task and rest. The system utilized two sessions dedicated solely to classifier training. Additionally, samples were collected prior to each period of online classification to update the classifier. Using a continuous-wave spectrometer, measurements were collected from the prefrontal and parietal cortices while 11 able-bodied adult participants were cued to perform one of the two cognitive tasks or rests. Each task was used to indicate the desire to select a particular letter on a scanning interface, while rest avoided selection. Classification was performed using 25 iteration of bagging with a linear discriminant base classifier. Classifiers were trained on 10-dimensional feature sets. The BCI's classification decision was provided as feedback. An average online classification accuracy of [Formula: see text]% was achieved, representing an ITR of [Formula: see text] bits/min. The results demonstrate that online communication can be achieved with a ternary NIRS-BCI that supports VFT, Stroop task and rest. Our findings encourage continued efforts to enhance the ITR of NIRS-BCIs.
Collapse
Affiliation(s)
- Larissa C Schudlo
- 1 Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, 150 Kilgour Road, Toronto, ON, Canada.,2 Institute of Biomaterial and Biomedical Engineering, University of Toronto, 164 College Street, Toronto, Ontario, Canada
| | - Tom Chau
- 1 Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital, 150 Kilgour Road, Toronto, ON, Canada.,2 Institute of Biomaterial and Biomedical Engineering, University of Toronto, 164 College Street, Toronto, Ontario, Canada
| |
Collapse
|
11
|
Hong KS, Khan MJ. Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review. Front Neurorobot 2017; 11:35. [PMID: 28790910 PMCID: PMC5522881 DOI: 10.3389/fnbot.2017.00035] [Citation(s) in RCA: 116] [Impact Index Per Article: 16.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2017] [Accepted: 07/03/2017] [Indexed: 12/11/2022] Open
Abstract
In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.
Collapse
Affiliation(s)
- Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Busan, South Korea.,Department of Cogno-Mechatronics Engineering, Pusan National University, Busan, South Korea
| | - Muhammad Jawad Khan
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
| |
Collapse
|
12
|
Khan MJ, Hong KS. Hybrid EEG-fNIRS-Based Eight-Command Decoding for BCI: Application to Quadcopter Control. Front Neurorobot 2017; 11:6. [PMID: 28261084 PMCID: PMC5314821 DOI: 10.3389/fnbot.2017.00006] [Citation(s) in RCA: 99] [Impact Index Per Article: 14.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2016] [Accepted: 01/24/2017] [Indexed: 01/27/2023] Open
Abstract
In this paper, a hybrid electroencephalography–functional near-infrared spectroscopy (EEG–fNIRS) scheme to decode eight active brain commands from the frontal brain region for brain–computer interface is presented. A total of eight commands are decoded by fNIRS, as positioned on the prefrontal cortex, and by EEG, around the frontal, parietal, and visual cortices. Mental arithmetic, mental counting, mental rotation, and word formation tasks are decoded with fNIRS, in which the selected features for classification and command generation are the peak, minimum, and mean ΔHbO values within a 2-s moving window. In the case of EEG, two eyeblinks, three eyeblinks, and eye movement in the up/down and left/right directions are used for four-command generation. The features in this case are the number of peaks and the mean of the EEG signal during 1 s window. We tested the generated commands on a quadcopter in an open space. An average accuracy of 75.6% was achieved with fNIRS for four-command decoding and 86% with EEG for another four-command decoding. The testing results show the possibility of controlling a quadcopter online and in real-time using eight commands from the prefrontal and frontal cortices via the proposed hybrid EEG–fNIRS interface.
Collapse
Affiliation(s)
- Muhammad Jawad Khan
- School of Mechanical Engineering, Pusan National University , Busan , South Korea
| | - Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Busan, South Korea; Department of Cogno-Mechatronics Engineering, Pusan National University, Busan, South Korea
| |
Collapse
|
13
|
Weyand S, Chau T. Challenges of implementing a personalized mental task near-infrared spectroscopy brain-computer interface for a non-verbal young adult with motor impairments. Dev Neurorehabil 2017; 20:99-107. [PMID: 26457507 DOI: 10.3109/17518423.2015.1087436] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/13/2022]
Abstract
PURPOSE Near-infrared spectroscopy brain-computer interfaces (NIRS-BCIs) have been proposed as potential motor-free communication pathways. This paper documents the challenges of implementing an NIRS-BCI with a non-verbal, severely and congenitally impaired, but cognitively intact young adult. METHODS A 5-session personalized mental task NIRS-BCI training paradigm was invoked, whereby participant-specific mental tasks were selected either by the researcher or by the user, on the basis of prior performance or user preference. RESULTS Although the personalized mental task selection and training framework had been previously demonstrated with able-bodied participants, the participant was not able to exceed chance-level accuracies. Challenges to the acquisition of BCI control may have included disinclination to BCI training, structural or functional brain atypicalities, heightened emotional arousal and confounding haemodynamic patterns associated with novelty and reward processing. CONCLUSIONS Overall, we stress the necessity for further clinical NIRS-BCI research involving non-verbal individuals with severe motor impairments.
Collapse
Affiliation(s)
- Sabine Weyand
- a Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital , Toronto , Ontario , Canada and.,b Institute of Biomaterials and Biomedical Engineering, University of Toronto , Ontario , Canada
| | - Tom Chau
- a Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital , Toronto , Ontario , Canada and.,b Institute of Biomaterials and Biomedical Engineering, University of Toronto , Ontario , Canada
| |
Collapse
|
14
|
Zafar A, Hong KS. Detection and classification of three-class initial dips from prefrontal cortex. BIOMEDICAL OPTICS EXPRESS 2017; 8:367-383. [PMID: 28101424 PMCID: PMC5231305 DOI: 10.1364/boe.8.000367] [Citation(s) in RCA: 63] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/29/2016] [Revised: 11/20/2016] [Accepted: 12/12/2016] [Indexed: 05/03/2023]
Abstract
In this paper, the use of initial dips using functional near-infrared spectroscopy (fNIRS) for brain-computer interface (BCI) is investigated. Features and window sizes for detecting initial dips are also discussed. Three mental tasks including mental arithmetic, mental counting, and puzzle solving are performed in obtaining fNIRS signals from the prefrontal cortex. Vector-based phase analysis method combined with a threshold circle, as a decision criterion, are used to detect the initial dips. Eight healthy subjects participate in experiment. Linear discriminant analysis is used as a classifier. To classify initial dips, five features (signal mean, peak value, signal slope, skewness, and kurtosis) of oxy-hemoglobin (HbO) and four different window sizes (0~1, 0~1.5, 0~2, and 0~2.5 sec) are examined. It is shown that a combination of signal mean and peak value and a time period of 0~2.5 sec provide the best average classification accuracy of 57.5% for three classes. To further validate the result, three-class classification using the conventional hemodynamic response (HR) is also performed, in which two features (signal mean and signal slope) and 2~7 sec window size have yielded the average classification accuracy of 65.9%. This reveals that fNIRS-based BCI using initial dip detection can reduce the command generation time from 7 sec to 2.5 sec while the classification accuracy is a bit sacrificed from 65.9% to 57.5% for three mental tasks. Further improvement can be made by using deoxy hemoglobin signals in coping with the slow HR problem.
Collapse
Affiliation(s)
- Amad Zafar
- School of Mechanical Engineering, Pusan National University, 2 Busandaehak-ro, Geumjeong-gu, Busan 46241, South Korea
| | - Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, 2 Busandaehak-ro, Geumjeong-gu, Busan 46241, South Korea
- Department of Cogno-Mechatronics Engineering, Pusan National University, 2 Busandaehak-ro, Geumjeong-gu, Busan 46241, South Korea
| |
Collapse
|
15
|
A Self-Paced P300 Healthcare Brain-Computer Interface System with SSVEP-Based Switching Control and Kernel FDA + SVM-Based Detector. APPLIED SCIENCES-BASEL 2016. [DOI: 10.3390/app6050142] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
|
16
|
Weyand S, Takehara-Nishiuchi K, Chau T. Exploring methodological frameworks for a mental task-based near-infrared spectroscopy brain–computer interface. J Neurosci Methods 2015. [DOI: 10.1016/j.jneumeth.2015.07.007] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
17
|
Weyand S, Chau T. Correlates of Near-Infrared Spectroscopy Brain-Computer Interface Accuracy in a Multi-Class Personalization Framework. Front Hum Neurosci 2015; 9:536. [PMID: 26483657 PMCID: PMC4588107 DOI: 10.3389/fnhum.2015.00536] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2015] [Accepted: 09/14/2015] [Indexed: 11/13/2022] Open
Abstract
Brain–computer interfaces (BCIs) provide individuals with a means of interacting with a computer using only neural activity. To date, the majority of near-infrared spectroscopy (NIRS) BCIs have used prescribed tasks to achieve binary control. The goals of this study were to evaluate the possibility of using a personalized approach to establish control of a two-, three-, four-, and five-class NIRS–BCI, and to explore how various user characteristics correlate to accuracy. Ten able-bodied participants were recruited for five data collection sessions. Participants performed six mental tasks and a personalized approach was used to select each individual’s best discriminating subset of tasks. The average offline cross-validation accuracies achieved were 78, 61, 47, and 37% for the two-, three-, four-, and five-class problems, respectively. Most notably, all participants exceeded an accuracy of 70% for the two-class problem, and two participants exceeded an accuracy of 70% for the three-class problem. Additionally, accuracy was found to be strongly positively correlated (Pearson’s) with perceived ease of session (ρ = 0.653), ease of concentration (ρ = 0.634), and enjoyment (ρ = 0.550), but strongly negatively correlated with verbal IQ (ρ = −0.749).
Collapse
Affiliation(s)
- Sabine Weyand
- PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital , Toronto, ON , Canada ; PRISM Laboratory, Institute of Biomaterials and Biomedical Engineering, University of Toronto , Toronto, ON , Canada
| | - Tom Chau
- PRISM Laboratory, Bloorview Research Institute, Holland Bloorview Kids Rehabilitation Hospital , Toronto, ON , Canada ; PRISM Laboratory, Institute of Biomaterials and Biomedical Engineering, University of Toronto , Toronto, ON , Canada
| |
Collapse
|