1
|
Eldawlatly S. On the role of generative artificial intelligence in the development of brain-computer interfaces. BMC Biomed Eng 2024; 6:4. [PMID: 38698495 PMCID: PMC11064240 DOI: 10.1186/s42490-024-00080-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2023] [Accepted: 04/24/2024] [Indexed: 05/05/2024] Open
Abstract
Since their inception more than 50 years ago, Brain-Computer Interfaces (BCIs) have held promise to compensate for functions lost by people with disabilities through allowing direct communication between the brain and external devices. While research throughout the past decades has demonstrated the feasibility of BCI to act as a successful assistive technology, the widespread use of BCI outside the lab is still beyond reach. This can be attributed to a number of challenges that need to be addressed for BCI to be of practical use including limited data availability, limited temporal and spatial resolutions of brain signals recorded non-invasively and inter-subject variability. In addition, for a very long time, BCI development has been mainly confined to specific simple brain patterns, while developing other BCI applications relying on complex brain patterns has been proven infeasible. Generative Artificial Intelligence (GAI) has recently emerged as an artificial intelligence domain in which trained models can be used to generate new data with properties resembling that of available data. Given the enhancements observed in other domains that possess similar challenges to BCI development, GAI has been recently employed in a multitude of BCI development applications to generate synthetic brain activity; thereby, augmenting the recorded brain activity. Here, a brief review of the recent adoption of GAI techniques to overcome the aforementioned BCI challenges is provided demonstrating the enhancements achieved using GAI techniques in augmenting limited EEG data, enhancing the spatiotemporal resolution of recorded EEG data, enhancing cross-subject performance of BCI systems and implementing end-to-end BCI applications. GAI could represent the means by which BCI would be transformed into a prevalent assistive technology, thereby improving the quality of life of people with disabilities, and helping in adopting BCI as an emerging human-computer interaction technology for general use.
Collapse
Affiliation(s)
- Seif Eldawlatly
- Computer and Systems Engineering Department, Faculty of Engineering, Ain Shams University, 1 El-Sarayat St., Abbassia, Cairo, Egypt.
- Computer Science and Engineering Department, The American University in Cairo, Cairo, Egypt.
| |
Collapse
|
2
|
Han J, Xu M, Xiao X, Yi W, Jung TP, Ming D. A high-speed hybrid brain-computer interface with more than 200 targets. J Neural Eng 2023; 20:016025. [PMID: 36608342 DOI: 10.1088/1741-2552/acb105] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Accepted: 01/06/2023] [Indexed: 01/07/2023]
Abstract
Objective. Brain-computer interfaces (BCIs) have recently made significant strides in expanding their instruction set, which has attracted wide attention from researchers. The number of targets and commands is a key indicator of how well BCIs can decode the brain's intentions. No studies have reported a BCI system with over 200 targets.Approach. This study developed the first high-speed BCI system with up to 216 targets that were encoded by a combination of electroencephalography features, including P300, motion visual evoked potential (mVEP), and steady-state visual evoked potential (SSVEP). Specifically, the hybrid BCI paradigm used the time-frequency division multiple access strategy to elaborately tag targets with P300 and mVEP of different time windows, along with SSVEP of different frequencies. The hybrid features were then decoded by task-discriminant component analysis and linear discriminant analysis. Ten subjects participated in the offline and online cued-guided spelling experiments. Other ten subjects took part in online free-spelling experiments.Main results.The offline results showed that the mVEP and P300 components were prominent in the central, parietal, and occipital regions, while the most distinct SSVEP feature was in the occipital region. The online cued-guided spelling and free-spelling results showed that the proposed BCI system achieved an average accuracy of 85.37% ± 7.49% and 86.00% ± 5.98% for the 216-target classification, resulting in an average information transfer rate (ITR) of 302.83 ± 39.20 bits min-1and 204.47 ± 37.56 bits min-1, respectively. Notably, the peak ITR could reach up to 367.83 bits min-1.Significance.This study developed the first high-speed BCI system with more than 200 targets, which holds promise for extending BCI's application scenarios.
Collapse
Affiliation(s)
- Jin Han
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
| | - Minpeng Xu
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| | - Xiaolin Xiao
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| | - Weibo Yi
- Beijing Machine and Equipment Institute, Beijing 100854, People's Republic of China
| | - Tzyy-Ping Jung
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Swartz Centre for Computational Neuroscience, University of California, San Diego, CA, United States of America
| | - Dong Ming
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, People's Republic of China
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, People's Republic of China
| |
Collapse
|
3
|
Wading corvus optimization based text generation using deep CNN and BiLSTM classifiers. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103969] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022]
|
4
|
An innovative P300 speller brain–computer interface design: Easy screen. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
|
5
|
Loizidou P, Rios E, Marttini A, Keluo-Udeke O, Soetedjo J, Belay J, Perifanos K, Pouratian N, Speier W. Extending Brain-Computer Interface Access with a Multilingual Language Model in the P300 Speller. BRAIN-COMPUTER INTERFACES 2022; 9:36-48. [PMID: 35574291 PMCID: PMC9094140 DOI: 10.1080/2326263x.2021.1993426] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
Abstract
Brain-computer interfaces (BCI) such as the P300 speller have the potential to restore communication to advanced-stage neuromuscular disease patients. Research has improved typing speed and accuracy through innovations including the use of language models. While significant advances have been made, implementations have largely been restricted to a single language, primarily English. It is unclear whether these improvements would extend to other languages that present potential technical hurdles due to different alphabets and grammatical structures. Here, we adapt a language model-based classifier designed for English to two other languages, Spanish and Greek, to demonstrate the generalizability of these methods. Online experimental trials with 30 healthy native English, Spanish, and Greek speakers showed no significant difference between performances using the different versions of the system (66.20 vs. 61.97 vs. 60.89 bits/minute). Extending these methods across languages allows for expanding access to BCI systems to other populations, particularly in the developing world.
Collapse
Affiliation(s)
- P Loizidou
- Radiological Sciences, University of California, Los Angeles, Los Angeles, CA 90024, USA
| | - E Rios
- Radiology, Stanford University, Stanford, CA 94305, USA
| | - A Marttini
- Radiological Sciences, University of California, Los Angeles, Los Angeles, CA 90024, USA
| | - O Keluo-Udeke
- Computer Science, University of Arkansas at Pine Bluff, Pine Bluff, AR 71601, USA
| | - J Soetedjo
- Bioengineering, University of Washington, Seattle, Washington 98195, USA
| | - J Belay
- Radiological Sciences, University of California, Los Angeles, Los Angeles, CA 90024, USA
| | - K Perifanos
- Linguistics, National and Kapodistrian University of Athens, Athens, Attica 15784, Greece
| | - N Pouratian
- Neurosurgery, University of California, Los Angeles, Los Angeles CA 90024, USA
| | - W Speier
- Radiological Sciences, University of California, Los Angeles, Los Angeles, CA 90024, USA,Corresponding Author: 924 Westwood Blvd, Suite 600, Los Angeles, CA 90024, (215) 206-6007,
| |
Collapse
|
6
|
Hill K, Huggins J, Woodworth C. Interprofessional Practitioners' Opinions on Features and Services for an Augmentative and Alternative Communication Brain-Computer Interface Device. PM R 2021; 13:1111-1121. [PMID: 33236859 PMCID: PMC10718316 DOI: 10.1002/pmrj.12525] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2020] [Revised: 10/08/2020] [Accepted: 11/16/2020] [Indexed: 11/06/2022]
Abstract
BACKGROUND Brain-computer interface (BCI) technology is an emerging access method to augmentative and alternative communication (AAC) devices. OBJECTIVES To identify, in the early stages of research and development, the perceptions and considerations of interprofessional practice (IPP) team members regarding features and functions for an AAC-BCI device. DESIGN Qualitative research methodology applying a grounded theory approach using focus groups with a follow-up survey of participants using NVivo analysis software supporting inductive coding of transcription data. SETTING Focus groups held at university, clinic, and industry conference rooms. Discussion was stimulated by a 14-minute video on an AAC-BCI device prototype. The prototype hardware and electroencephalography (EEG) gel and dry electrode headgear were on display. PARTICIPANTS Convenience sample of practitioners providing rehabilitation or clinical services to individuals with severe communication disorders and movement impairments who use AAC and/or other assistive technology. INTERVENTIONS Not applicable. MAIN OUTCOME MEASURES Descriptive statistics using thematic analysis of participants' opinions, input, and feedback on the ideal design for a noninvasive, EEG-based P300 AAC-BCI device. RESULTS Interrater and interjudge reliability were at 98% and 100%, respectively, for transcription and researcher coding. Triangulation of multiple data sources supported theme and subtheme identification that included design features, set-up and calibration, services, and effectiveness. An AAC device with BCI access was unanimously confirmed (100%) as a desirable commercial product. Participants felt that the AAC-BCI prototype appeared effective for meeting daily communication needs (75%). Results showed that participants' preference on headgear types would change based on accuracy (91%) and rate (83%) of performance. A data-logging feature was considered beneficial by 100% of participants. CONCLUSIONS IPP teams provided critical impressions on design, services, and features for a commercial AAC-BCI device. Expressed feature and function preferences showed dependence on communication accuracy, rate, and effectiveness. This provides vital guidance for successful clinical deployment.
Collapse
Affiliation(s)
- Katya Hill
- Department of Communication Science and Disorders, University of Pittsburgh, Pittsburgh, PA, USA
| | - Jane Huggins
- Departments of Physical Medicine and Rehabilitation and Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA
| | - Chelsea Woodworth
- Department of Communication Science and Disorders, University of Pittsburgh, Pittsburgh, PA, USA
| |
Collapse
|
7
|
Velasco-Álvarez F, Fernández-Rodríguez Á, Vizcaíno-Martín FJ, Díaz-Estrella A, Ron-Angevin R. Brain-Computer Interface (BCI) Control of a Virtual Assistant in a Smartphone to Manage Messaging Applications. SENSORS (BASEL, SWITZERLAND) 2021; 21:3716. [PMID: 34073602 PMCID: PMC8199460 DOI: 10.3390/s21113716] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/07/2021] [Revised: 05/21/2021] [Accepted: 05/25/2021] [Indexed: 12/13/2022]
Abstract
Brain-computer interfaces (BCI) are a type of assistive technology that uses the brain signals of users to establish a communication and control channel between them and an external device. BCI systems may be a suitable tool to restore communication skills in severely motor-disabled patients, as BCI do not rely on muscular control. The loss of communication is one of the most negative consequences reported by such patients. This paper presents a BCI system focused on the control of four mainstream messaging applications running in a smartphone: WhatsApp, Telegram, e-mail and short message service (SMS). The control of the BCI is achieved through the well-known visual P300 row-column paradigm (RCP), allowing the user to select control commands as well as spelling characters. For the control of the smartphone, the system sends synthesized voice commands that are interpreted by a virtual assistant running in the smartphone. Four tasks related to the four mentioned messaging services were tested with 15 healthy volunteers, most of whom were able to accomplish the tasks, which included sending free text e-mails to an address proposed by the subjects themselves. The online performance results obtained, as well as the results of subjective questionnaires, support the viability of the proposed system.
Collapse
Affiliation(s)
- Francisco Velasco-Álvarez
- Departamento de Tecnología Electrónica, Universidad de Málaga, 29071 Málaga, Spain; (Á.F.-R.); (F.-J.V.-M.); (A.D.-E.); (R.R.-A.)
| | | | | | | | | |
Collapse
|
8
|
Fernández-Rodríguez Á, Medina-Juliá MT, Velasco-Álvarez F, Ron-Angevin R. Different effects of using pictures as stimuli in a P300 brain-computer interface under rapid serial visual presentation or row-column paradigm. Med Biol Eng Comput 2021; 59:869-881. [PMID: 33742353 DOI: 10.1007/s11517-021-02340-y] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2020] [Accepted: 02/23/2021] [Indexed: 02/08/2023]
Abstract
Previous proposals for controlling a P300-based BCI speller have shown an improvement using alternative images instead of letters as target stimuli under a row-column paradigm (RCP). However, the RCP is not suitable for those patients with a lack of gaze control. To solve that, the rapid serial visual presentation (RSVP) paradigm has been proposed in previous studies. The aim of the present work is to assess if a set of alternative pictures that improved performance in RCP could also improve performance in RSVP. Sixteen participants controlled four conditions in calibration and online tasks: letters in RCP, pictures in RCP, letters in RSVP and pictures in RSVP. The effect given by pictures was greater under RCP than under RSVP, both for performance and event-related potential analyses. Indeed, pictures did not show any improvement under RSVP in comparison to letters. In addition, the condition with pictures under RCP was declared the favourite by most users (68.75%), while the condition with pictures under RSVP was not chosen as favourite by any participant. Therefore, this work shows that the improvement related to the use of pictures as alternative flashing stimuli under RCP may not be transferred to RSVP. Graphical abstract.
Collapse
Affiliation(s)
- Álvaro Fernández-Rodríguez
- Departamento de Tecnología Electrónica, Universidad de Málaga, 35 Louis Pasteur Boulevard, 29071, Malaga, Spain.
| | - María Teresa Medina-Juliá
- Departamento de Tecnología Electrónica, Universidad de Málaga, 35 Louis Pasteur Boulevard, 29071, Malaga, Spain
| | - Francisco Velasco-Álvarez
- Departamento de Tecnología Electrónica, Universidad de Málaga, 35 Louis Pasteur Boulevard, 29071, Malaga, Spain
| | - Ricardo Ron-Angevin
- Departamento de Tecnología Electrónica, Universidad de Málaga, 35 Louis Pasteur Boulevard, 29071, Malaga, Spain
| |
Collapse
|
9
|
Abstract
Hyperscanning is a technique which simultaneously records the neural activity of two or more people. This is done using one of several neuroimaging methods, such as electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The use of hyperscanning has seen a dramatic rise in recent years to monitor social interactions between two or more people. Similarly, there has been an increase in the use of virtual reality (VR) for collaboration, and an increase in the frequency of social interactions being carried out in virtual environments (VE). In light of this, it is important to understand how interactions function within VEs, and how they can be enhanced to improve their quality in a VE. In this paper, we present some of the work that has been undertaken in the field of social neuroscience, with a special emphasis on hyperscanning. We also cover the literature detailing the work that has been carried out in the human–computer interaction domain that addresses remote collaboration. Finally, we present a way forward where these two research domains can be combined to explore how monitoring the neural activity of a group of participants in VE could enhance collaboration among them.
Collapse
|
10
|
Gembler FW, Benda M, Rezeika A, Stawicki PR, Volosyak I. Asynchronous c-VEP communication tools-efficiency comparison of low-target, multi-target and dictionary-assisted BCI spellers. Sci Rep 2020; 10:17064. [PMID: 33051500 PMCID: PMC7553931 DOI: 10.1038/s41598-020-74143-4] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2020] [Accepted: 09/28/2020] [Indexed: 11/25/2022] Open
Abstract
Keyboards and smartphones allow users to express their thoughts freely via manual control. Hands-free communication can be realized with brain-computer interfaces (BCIs) based on code-modulated visual evoked potentials (c-VEPs). Various variations of such spellers have been developed: Low-target systems, multi-target systems and systems with dictionary support. In general, it is not clear which kinds of systems are optimal in terms of reliability, speed, cognitive load, and visual load. The presented study investigates the feasibility of different speller variations. 58 users tested a 4-target speller and a 32-target speller with and without dictionary functionality. For classification, multiple individualized spatial filters were generated via canonical correlation analysis (CCA). We used an asynchronous implementation allowing non-control state, thus aiming for high accuracy rather than speed. All users were able to control the tested spellers. Interestingly, no significant differences in accuracy were found: 94.4%, 95.5% and 94.0% for 4-target spelling, 32-target spelling, and dictionary-assisted 32-target spelling. The mean ITRs were highest for the 32-target interface: 45.2, 96.9 and 88.9 bit/min. The output speed in characters per minute, was highest in dictionary-assisted spelling: 8.2, 19.5 and 31.6 characters/min. According to questionnaire results, 86% of the participants preferred the 32-target speller over the 4-target speller.
Collapse
Affiliation(s)
- Felix W Gembler
- Rhine-Waal University of Applied Sciences, Technology and Bionics, 47533, Kleve, Germany
| | - Mihaly Benda
- Rhine-Waal University of Applied Sciences, Technology and Bionics, 47533, Kleve, Germany
| | - Aya Rezeika
- Rhine-Waal University of Applied Sciences, Technology and Bionics, 47533, Kleve, Germany
| | - Piotr R Stawicki
- Rhine-Waal University of Applied Sciences, Technology and Bionics, 47533, Kleve, Germany
| | - Ivan Volosyak
- Rhine-Waal University of Applied Sciences, Technology and Bionics, 47533, Kleve, Germany.
| |
Collapse
|
11
|
Kshirsagar GB, Londhe ND. Weighted Ensemble of Deep Convolution Neural Networks for Single-Trial Character Detection in Devanagari-Script-Based P300 Speller. IEEE Trans Cogn Dev Syst 2020. [DOI: 10.1109/tcds.2019.2942437] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
12
|
Ratcliffe L, Puthusserypady S. Importance of Graphical User Interface in the design of P300 based Brain–Computer Interface systems. Comput Biol Med 2020; 117:103599. [DOI: 10.1016/j.compbiomed.2019.103599] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2019] [Revised: 12/12/2019] [Accepted: 12/29/2019] [Indexed: 12/01/2022]
|
13
|
Renton AI, Mattingley JB, Painter DR. Optimising non-invasive brain-computer interface systems for free communication between naïve human participants. Sci Rep 2019; 9:18705. [PMID: 31822715 PMCID: PMC6904487 DOI: 10.1038/s41598-019-55166-y] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2019] [Accepted: 11/22/2019] [Indexed: 12/22/2022] Open
Abstract
Free communication is one of the cornerstones of modern civilisation. While manual keyboards currently allow us to interface with computers and manifest our thoughts, a next frontier is communication without manual input. Brain-computer interface (BCI) spellers often achieve this by decoding patterns of neural activity as users attend to flickering keyboard displays. To date, the highest performing spellers report typing rates of ~10.00 words/minute. While impressive, these rates are typically calculated for experienced users repetitively typing single phrases. It is therefore not clear whether naïve users are able to achieve such high rates with the added cognitive load of genuine free communication, which involves continuously generating and spelling novel words and phrases. In two experiments, we developed an open-source, high-performance, non-invasive BCI speller and examined its feasibility for free communication. The BCI speller required users to focus their visual attention on a flickering keyboard display, thereby producing unique cortical activity patterns for each key, which were decoded using filter-bank canonical correlation analysis. In Experiment 1, we tested whether seventeen naïve users could maintain rapid typing during prompted free word association. We found that information transfer rates were indeed slower during this free communication task than during typing of a cued character sequence. In Experiment 2, we further evaluated the speller's efficacy for free communication by developing a messaging interface, allowing users to engage in free conversation. The results showed that free communication was possible, but that information transfer was reduced by voluntary textual corrections and turn-taking during conversation. We evaluated a number of factors affecting the suitability of BCI spellers for free communication, and make specific recommendations for improving classification accuracy and usability. Overall, we found that developing a BCI speller for free communication requires a focus on usability over reduced character selection time, and as such, future performance appraisals should be based on genuine free communication scenarios.
Collapse
Affiliation(s)
- Angela I Renton
- Queensland Brain Institute, The University of Queensland, St Lucia, 4072, Australia.
| | - Jason B Mattingley
- Queensland Brain Institute, The University of Queensland, St Lucia, 4072, Australia
- School of Psychology, The University of Queensland, St Lucia, 4072, Australia
- Canadian Institute for Advanced Research (CIFAR), Toronto, Canada
| | - David R Painter
- School of Psychology, The University of Queensland, St Lucia, 4072, Australia
| |
Collapse
|
14
|
Jumphoo T, Uthansakul M, Uthansakul P. Brainwave classification without the help of limb movement and any stimulus for character-writing application. COGN SYST RES 2019. [DOI: 10.1016/j.cogsys.2019.09.002] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
15
|
Fernández-Rodríguez Á, Velasco-Álvarez F, Medina-Juliá MT, Ron-Angevin R. Evaluation of emotional and neutral pictures as flashing stimuli using a P300 brain-computer interface speller. J Neural Eng 2019; 16:056024. [PMID: 31382248 DOI: 10.1088/1741-2552/ab386d] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Previous works have reported that complex emotional and visual stimuli can increase the amplitude of the P300 brain potential. Thus, the aim of the present work is to assess these kinds of images in a P300 brain-computer interface (BCI) speller as flashing stimuli. APPROACH Twenty-three volunteers controlled four spellers with different sets of flashing stimuli: flashing letters, neutral pictures (NP), emotional pleasant pictures (EPP) and emotional unpleasant pictures (EUP). MAIN RESULTS The sets of pictures showed a higher performance than the letters in accuracy and information transfer rate. These results were supported by the analysis of the P300 signal, where the picture sets offered the greatest amplitudes. The NP and EPP sets were the best evaluated in the subjective questionnaire. SIGNIFICANCE In short, despite the fact that the effect of emotional stimuli could not be observed in the performance metrics, picture sets have offered a high performance and should be considered in future proposals for visual P300-based BCI applications.
Collapse
|
16
|
Rezeika A, Benda M, Stawicki P, Gembler F, Saboor A, Volosyak I. Brain-Computer Interface Spellers: A Review. Brain Sci 2018; 8:brainsci8040057. [PMID: 29601538 PMCID: PMC5924393 DOI: 10.3390/brainsci8040057] [Citation(s) in RCA: 145] [Impact Index Per Article: 24.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2018] [Revised: 03/16/2018] [Accepted: 03/27/2018] [Indexed: 12/14/2022] Open
Abstract
A Brain-Computer Interface (BCI) provides a novel non-muscular communication method via brain signals. A BCI-speller can be considered as one of the first published BCI applications and has opened the gate for many advances in the field. Although many BCI-spellers have been developed during the last few decades, to our knowledge, no reviews have described the different spellers proposed and studied in this vital field. The presented speller systems are categorized according to major BCI paradigms: P300, steady-state visual evoked potential (SSVEP), and motor imagery (MI). Different BCI paradigms require specific electroencephalogram (EEG) signal features and lead to the development of appropriate Graphical User Interfaces (GUIs). The purpose of this review is to consolidate the most successful BCI-spellers published since 2010, while mentioning some other older systems which were built explicitly for spelling purposes. We aim to assist researchers and concerned individuals in the field by illustrating the highlights of different spellers and presenting them in one review. It is almost impossible to carry out an objective comparison between different spellers, as each has its variables, parameters, and conditions. However, the gathered information and the provided taxonomy about different BCI-spellers can be helpful, as it could identify suitable systems for first-hand users, as well as opportunities of development and learning from previous studies for BCI researchers.
Collapse
Affiliation(s)
- Aya Rezeika
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Mihaly Benda
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Piotr Stawicki
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Felix Gembler
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Abdul Saboor
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Ivan Volosyak
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| |
Collapse
|
17
|
Speier W, Arnold C, Chandravadia N, Roberts D, Pendekanti S, Pouratian N. Improving P300 Spelling Rate using Language Models and Predictive Spelling. BRAIN-COMPUTER INTERFACES 2017; 5:13-22. [PMID: 30560145 DOI: 10.1080/2326263x.2017.1410418] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Abstract
The P300 Speller Brain-Computer Interface (BCI) provides a means of communication for those suffering from advanced neuromuscular diseases such as amyotrophic lateral sclerosis (ALS). Recent literature has incorporated language-based modelling, which uses previously chosen characters and the structure of natural language to modify the interface and classifier. Two complementary methods of incorporating language models have previously been independently studied: predictive spelling uses language models to generate suggestions of complete words to allow for the selection of multiple characters simultaneously, and language model-based classifiers have used prior characters to create a prior probability distribution over the characters based on how likely they are to follow. In this study, we propose a combined method which extends a language-based classifier to generate prior probabilities for both individual characters and complete words. In order to gauge the efficiency of this new model, results across 12 healthy subjects were measured. Incorporating predictive spelling increased typing speed using the P300 speller, with an average increase of 15.5% in typing rate across subjects, demonstrating that language models can be effectively utilized to create full word suggestions for predictive spelling. When combining predictive spelling with language model classification, typing speed is significantly improved, resulting in better typing performance.
Collapse
Affiliation(s)
- William Speier
- Department of Neurosurgery, University of California, Los Angeles, USA.,Medical Imaging Informatics Group, University of California, Los Angeles, USA
| | - Corey Arnold
- Medical Imaging Informatics Group, University of California, Los Angeles, USA
| | - Nand Chandravadia
- Neuroscience Interdepartmental Program, University of California, Los Angeles, USA
| | - Dustin Roberts
- Department of Neurosurgery, University of California, Los Angeles, USA
| | - Shrita Pendekanti
- Neuroscience Interdepartmental Program, University of California, Los Angeles, USA
| | - Nader Pouratian
- Department of Neurosurgery, University of California, Los Angeles, USA.,Neuroscience Interdepartmental Program, University of California, Los Angeles, USA.,Department of Bioengineering, University of California, Los Angeles, USA.,Brain Research Institute, University of California, Los Angeles, USA
| |
Collapse
|
18
|
Ryan DB, Townsend G, Gates NA, Colwell K, Sellers EW. Evaluating brain-computer interface performance using color in the P300 checkerboard speller. Clin Neurophysiol 2017; 128:2050-2057. [PMID: 28863361 DOI: 10.1016/j.clinph.2017.07.397] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2015] [Revised: 07/12/2017] [Accepted: 07/14/2017] [Indexed: 11/18/2022]
Abstract
OBJECTIVE Current Brain-Computer Interface (BCI) systems typically flash an array of items from grey to white (GW). The objective of this study was to evaluate BCI performance using uniquely colored stimuli. METHODS In addition to the GW stimuli, the current study tested two types of color stimuli (grey to color [GC] and color intensification [CI]). The main hypotheses were that in a checkboard paradigm, unique color stimuli will: (1) increase BCI performance over the standard GW paradigm; (2) elicit larger event-related potentials (ERPs); and, (3) improve offline performance with an electrode selection algorithm (i.e., Jumpwise). RESULTS Online results (n=36) showed that GC provides higher accuracy and information transfer rate than the CI and GW conditions. Waveform analysis showed that GC produced higher amplitude ERPs than CI and GW. Information transfer rate was improved by the Jumpwise-selected channel locations in all conditions. CONCLUSIONS Unique color stimuli (GC) improved BCI performance and enhanced ERPs. Jumpwise-selected electrode locations improved offline performance. SIGNIFICANCE These results show that in a checkerboard paradigm, unique color stimuli increase BCI performance, are preferred by participants, and are important to the design of end-user applications; thus, could lead to an increase in end-user performance and acceptance of BCI technology.
Collapse
Affiliation(s)
- D B Ryan
- Department of Psychology, East Tennessee State University, Johnson City, TN, USA.
| | - G Townsend
- Department of Computer Science, Algoma University, Sault Ste. Marie, Ontario, Canada
| | - N A Gates
- Department of Psychology, East Tennessee State University, Johnson City, TN, USA
| | - K Colwell
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA
| | - E W Sellers
- Department of Psychology, East Tennessee State University, Johnson City, TN, USA
| |
Collapse
|
19
|
Iljina O, Derix J, Schirrmeister RT, Schulze-Bonhage A, Auer P, Aertsen A, Ball T. Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication. BRAIN-COMPUTER INTERFACES 2017. [DOI: 10.1080/2326263x.2017.1330611] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
- Olga Iljina
- GRK 1624 ‘Frequency effects in language’, University of Freiburg, Freiburg, Germany
- Department of German Linguistics, University of Freiburg, Freiburg, Germany
- Hermann Paul School of Linguistics, University of Freiburg, Germany
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
- Neurobiology and Biophysics, Faculty of Biology, University of Freiburg, Freiburg, Germany
| | - Johanna Derix
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
- Translational Neurotechnology Lab, Department of Neurosurgery, University Medical Center Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Robin Tibor Schirrmeister
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
- Translational Neurotechnology Lab, Department of Neurosurgery, University Medical Center Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Andreas Schulze-Bonhage
- Epilepsy Center, Department of Neurosurgery, University Medical Center Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
| | - Peter Auer
- GRK 1624 ‘Frequency effects in language’, University of Freiburg, Freiburg, Germany
- Department of German Linguistics, University of Freiburg, Freiburg, Germany
- Hermann Paul School of Linguistics, University of Freiburg, Germany
- Freiburg Institute for Advanced Studies (FRIAS), University of Freiburg, Freiburg, Germany
| | - Ad Aertsen
- Neurobiology and Biophysics, Faculty of Biology, University of Freiburg, Freiburg, Germany
- Bernstein Center Freiburg, University of Freiburg, Germany
| | - Tonio Ball
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
- Translational Neurotechnology Lab, Department of Neurosurgery, University Medical Center Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| |
Collapse
|
20
|
Gonzalez-Navarro P, Moghadamfalahi M, Akcakaya M, Erdogmus D. Spatio-Temporal EEG Models for Brain Interfaces. SIGNAL PROCESSING 2017; 131:333-343. [PMID: 27713590 PMCID: PMC5047025 DOI: 10.1016/j.sigpro.2016.08.001] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
Multichannel electroencephalography (EEG) is widely used in non-invasive brain computer interfaces (BCIs) for user intent inference. EEG can be assumed to be a Gaussian process with unknown mean and autocovariance, and the estimation of parameters is required for BCI inference. However, the relatively high dimensionality of the EEG feature vectors with respect to the number of labeled observations lead to rank deficient covariance matrix estimates. In this manuscript, to overcome ill-conditioned covariance estimation, we propose a structure for the covariance matrices of the multichannel EEG signals. Specifically, we assume that these covariances can be modeled as a Kronecker product of temporal and spatial covariances. Our results over the experimental data collected from the users of a letter-by-letter typing BCI show that with less number of parameter estimations, the system can achieve higher classification accuracies compared to a method that uses full unstructured covariance estimation. Moreover, in order to illustrate that the proposed Kronecker product structure could enable shortening the BCI calibration data collection sessions, using Cramer-Rao bound analysis on simulated data, we demonstrate that a model with structured covariance matrices will achieve the same estimation error as a model with no covariance structure using fewer labeled EEG observations.
Collapse
Affiliation(s)
| | | | - M Akcakaya
- University of Pittsburgh, Pittsburgh, PA
| | | |
Collapse
|
21
|
Lee J, Whang M, Yoon J, Park M, Kim J. Optimized inter-stimulus interval (ISI) and content design for evoking better visual evoked potential (VEP) in brain-computer interface applications. BRAIN-COMPUTER INTERFACES 2016. [DOI: 10.1080/2326263x.2016.1253524] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Affiliation(s)
- Jungnyun Lee
- Emotion Contents Technology Research Center, Sangmyung University, Hongji-dong, Jongno-gu, Seoul, Korea
| | - Mincheol Whang
- Department of Media Software, Sangmyung University, Hongji-dong, Jongno-gu, Seoul, Korea
| | - Jaehong Yoon
- Emotion Contents Technology Research Center, Sangmyung University, Hongji-dong, Jongno-gu, Seoul, Korea
| | - Minji Park
- Emotion Contents Technology Research Center, Sangmyung University, Hongji-dong, Jongno-gu, Seoul, Korea
| | - Jonghwa Kim
- Research Associate, VR/AR center, Korea Electronics Technology Institute Digital Innovation Center
| |
Collapse
|
22
|
Speier W, Arnold C, Pouratian N. Integrating language models into classifiers for BCI communication: a review. J Neural Eng 2016; 13:031002. [PMID: 27153565 DOI: 10.1088/1741-2560/13/3/031002] [Citation(s) in RCA: 32] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE The present review systematically examines the integration of language models to improve classifier performance in brain-computer interface (BCI) communication systems. APPROACH The domain of natural language has been studied extensively in linguistics and has been used in the natural language processing field in applications including information extraction, machine translation, and speech recognition. While these methods have been used for years in traditional augmentative and assistive communication devices, information about the output domain has largely been ignored in BCI communication systems. Over the last few years, BCI communication systems have started to leverage this information through the inclusion of language models. MAIN RESULTS Although this movement began only recently, studies have already shown the potential of language integration in BCI communication and it has become a growing field in BCI research. BCI communication systems using language models in their classifiers have progressed down several parallel paths, including: word completion; signal classification; integration of process models; dynamic stopping; unsupervised learning; error correction; and evaluation. SIGNIFICANCE Each of these methods have shown significant progress, but have largely been addressed separately. Combining these methods could use the full potential of language model, yielding further performance improvements. This integration should be a priority as the field works to create a BCI system that meets the needs of the amyotrophic lateral sclerosis population.
Collapse
Affiliation(s)
- W Speier
- Department of Neurosurgery, University of California, Los Angeles, CA 90095, USA. Medical Imaging Informatics Group, University of California, Los Angeles, CA 90095, USA
| | | | | |
Collapse
|
23
|
Orhan U, Nezamfar H, Akcakaya M, Erdogmus D, Higger M, Moghadamfalahi M, Fowler A, Roark B, Oken B, Fried-Oken M. Probabilistic Simulation Framework for EEG-Based BCI Design. BRAIN-COMPUTER INTERFACES 2016; 3:171-185. [PMID: 29250562 DOI: 10.1080/2326263x.2016.1252621] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
A simulation framework could decrease the burden of attending long and tiring experimental sessions on the potential users of brain computer interface (BCI) systems. Specifically during the initial design of a BCI, a simulation framework that could replicate the operational performance of the system would be a useful tool for designers to make design choices. In this manuscript, we develop a Monte Carlo based probabilistic simulation framework for electroencephalography (EEG) based BCI design. We employ one event related potential (ERP) based typing and one steady state evoked potential (SSVEP) based control interface as testbeds. We compare the results of simulations with real time experiments. Even though over and under estimation of the performance is possible, the statistical results over the Monte Carlo simulations show that the developed framework generally provides a good approximation of the real time system performance.
Collapse
|
24
|
Moghadamfalahi M, Orhan U, Akcakaya M, Nezamfar H, Fried-Oken M, Erdogmus D. Language-Model Assisted Brain Computer Interface for Typing: A Comparison of Matrix and Rapid Serial Visual Presentation. IEEE Trans Neural Syst Rehabil Eng 2015; 23:910-20. [DOI: 10.1109/tnsre.2015.2411574] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
25
|
Ron-Angevin R, Varona-Moya S, Silva-Sauer LD. Initial test of a T9-like P300-based speller by an ALS patient. J Neural Eng 2015; 12:046023. [DOI: 10.1088/1741-2560/12/4/046023] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
|
26
|
Speier W, Arnold CW, Deshpande A, Knall J, Pouratian N. Incorporating advanced language models into the P300 speller using particle filtering. J Neural Eng 2015; 12:046018. [PMID: 26061188 DOI: 10.1088/1741-2560/12/4/046018] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
OBJECTIVE The P300 speller is a common brain-computer interface (BCI) application designed to communicate language by detecting event related potentials in a subject's electroencephalogram signal. Information about the structure of natural language can be valuable for BCI communication, but attempts to use this information have thus far been limited to rudimentary n-gram models. While more sophisticated language models are prevalent in natural language processing literature, current BCI analysis methods based on dynamic programming cannot handle their complexity. APPROACH Sampling methods can overcome this complexity by estimating the posterior distribution without searching the entire state space of the model. In this study, we implement sequential importance resampling, a commonly used particle filtering (PF) algorithm, to integrate a probabilistic automaton language model. MAIN RESULT This method was first evaluated offline on a dataset of 15 healthy subjects, which showed significant increases in speed and accuracy when compared to standard classification methods as well as a recently published approach using a hidden Markov model (HMM). An online pilot study verified these results as the average speed and accuracy achieved using the PF method was significantly higher than that using the HMM method. SIGNIFICANCE These findings strongly support the integration of domain-specific knowledge into BCI classification to improve system performance.
Collapse
Affiliation(s)
- W Speier
- Department of Bioengineering, University of California, Los Angeles, CA 90095, USA
| | | | | | | | | |
Collapse
|
27
|
Mainsah BO, Collins LM, Colwell KA, Sellers EW, Ryan DB, Caves K, Throckmorton CS. Increasing BCI communication rates with dynamic stopping towards more practical use: an ALS study. J Neural Eng 2015; 12:016013. [PMID: 25588137 DOI: 10.1088/1741-2560/12/1/016013] [Citation(s) in RCA: 42] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE The P300 speller is a brain-computer interface (BCI) that can possibly restore communication abilities to individuals with severe neuromuscular disabilities, such as amyotrophic lateral sclerosis (ALS), by exploiting elicited brain signals in electroencephalography (EEG) data. However, accurate spelling with BCIs is slow due to the need to average data over multiple trials to increase the signal-to-noise ratio (SNR) of the elicited brain signals. Probabilistic approaches to dynamically control data collection have shown improved performance in non-disabled populations; however, validation of these approaches in a target BCI user population has not occurred. APPROACH We have developed a data-driven algorithm for the P300 speller based on Bayesian inference that improves spelling time by adaptively selecting the number of trials based on the acute SNR of a user's EEG data. We further enhanced the algorithm by incorporating information about the user's language. In this current study, we test and validate the algorithms online in a target BCI user population, by comparing the performance of the dynamic stopping (DS) (or early stopping) algorithms against the current state-of-the-art method, static data collection, where the amount of data collected is fixed prior to online operation. MAIN RESULTS Results from online testing of the DS algorithms in participants with ALS demonstrate a significant increase in communication rate as measured in bits/min (100-300%), and theoretical bit rate (100-550%), while maintaining selection accuracy. Participants also overwhelmingly preferred the DS algorithms. SIGNIFICANCE We have developed a viable BCI algorithm that has been tested in a target BCI population which has the potential for translation to improve BCI speller performance towards more practical use for communication.
Collapse
Affiliation(s)
- B O Mainsah
- Duke University, Department of Electrical and Computer Engineering, USA
| | | | | | | | | | | | | |
Collapse
|
28
|
An efficient word typing P300-BCI system using a modified T9 interface and random forest classifier. Comput Biol Med 2015; 56:30-6. [DOI: 10.1016/j.compbiomed.2014.10.021] [Citation(s) in RCA: 56] [Impact Index Per Article: 6.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2014] [Revised: 10/21/2014] [Accepted: 10/25/2014] [Indexed: 11/30/2022]
|
29
|
Fried-Oken M, Mooney A, Peters B. Supporting communication for patients with neurodegenerative disease. NeuroRehabilitation 2015; 37:69-87. [PMID: 26409694 PMCID: PMC6380499 DOI: 10.3233/nre-151241] [Citation(s) in RCA: 60] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
BACKGROUND Communication supports, referred to as augmentative and alternative communication (AAC), are an integral part of medical speech-language pathology practice, yet many providers remain unfamiliar with assessment and intervention principles. For patients with complex communication impairments secondary to neurodegenerative disease, AAC services differ depending on whether their condition primarily affects speech and motor skills (ALS), language (primary progressive aphasia) or cognition (Alzheimer's disease). This review discusses symptom management for these three conditions, identifying behavioral strategies, low- and high-tech solutions for implementation during the natural course of disease. These AAC principles apply to all neurodegenerative diseases in which common symptoms appear. OBJECTIVES To present AAC interventions for patients with neurodegenerative diseases affecting speech, motor, language and cognitive domains. Three themes emerge: (1) timing of intervention: early referral, regular re-evaluations and continual treatment are essential; (2) communication partners must be included from the onset to establish AAC acceptance and use; and (3) strategies will change over time and use multiple modalities to capitalize on patients' strengths. CONCLUSIONS AAC should be standard practice for adults with neurodegenerative disease. Patients can maintain effective, functional communication with AAC supports. Individualized communication systems can be implemented ensuring patients remain active participants in daily activities.
Collapse
|
30
|
Bacher D, Jarosiewicz B, Masse NY, Stavisky SD, Simeral JD, Newell K, Oakley EM, Cash SS, Friehs G, Hochberg LR. Neural Point-and-Click Communication by a Person With Incomplete Locked-In Syndrome. Neurorehabil Neural Repair 2014; 29:462-71. [PMID: 25385765 DOI: 10.1177/1545968314554624] [Citation(s) in RCA: 64] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
A goal of brain-computer interface research is to develop fast and reliable means of communication for individuals with paralysis and anarthria. We evaluated the ability of an individual with incomplete locked-in syndrome enrolled in the BrainGate Neural Interface System pilot clinical trial to communicate using neural point-and-click control. A general-purpose interface was developed to provide control of a computer cursor in tandem with one of two on-screen virtual keyboards. The novel BrainGate Radial Keyboard was compared to a standard QWERTY keyboard in a balanced copy-spelling task. The Radial Keyboard yielded a significant improvement in typing accuracy and speed-enabling typing rates over 10 correct characters per minute. The participant used this interface to communicate face-to-face with research staff by using text-to-speech conversion, and remotely using an internet chat application. This study demonstrates the first use of an intracortical brain-computer interface for neural point-and-click communication by an individual with incomplete locked-in syndrome.
Collapse
Affiliation(s)
- Daniel Bacher
- School of Engineering, Brown University, Providence, RI, USA Institute for Brain Science, Brown University, Providence, RI, USA
| | - Beata Jarosiewicz
- Dept. of Neuroscience, Brown University, Providence, RI, USA Institute for Brain Science, Brown University, Providence, RI, USA Center for Neurorestoration and Neurotechnology, Dept. Veterans Affairs Medical Center, Providence, RI, USA
| | - Nicolas Y Masse
- Dept. of Neuroscience, Brown University, Providence, RI, USA Institute for Brain Science, Brown University, Providence, RI, USA
| | - Sergey D Stavisky
- School of Engineering, Brown University, Providence, RI, USA Center for Neurorestoration and Neurotechnology, Dept. Veterans Affairs Medical Center, Providence, RI, USA
| | - John D Simeral
- School of Engineering, Brown University, Providence, RI, USA Institute for Brain Science, Brown University, Providence, RI, USA Center for Neurorestoration and Neurotechnology, Dept. Veterans Affairs Medical Center, Providence, RI, USA Dept. of Neurology, Massachusetts General Hospital, Boston, MA, USA
| | - Katherine Newell
- Dept. of Neurology, Massachusetts General Hospital, Boston, MA, USA
| | - Erin M Oakley
- Dept. of Neurology, Massachusetts General Hospital, Boston, MA, USA
| | - Sydney S Cash
- Dept. of Neurology, Massachusetts General Hospital, Boston, MA, USA Dept. of Neurology, Harvard Medical School, Boston, MA, USA
| | - Gerhard Friehs
- Dept. of Neurosurgery, Rhode Island Hospital, Providence, RI, USA
| | - Leigh R Hochberg
- School of Engineering, Brown University, Providence, RI, USA Institute for Brain Science, Brown University, Providence, RI, USA Center for Neurorestoration and Neurotechnology, Dept. Veterans Affairs Medical Center, Providence, RI, USA Dept. of Neurology, Massachusetts General Hospital, Boston, MA, USA Dept. of Neurology, Harvard Medical School, Boston, MA, USA
| |
Collapse
|
31
|
Speier W, Deshpande A, Pouratian N. A method for optimizing EEG electrode number and configuration for signal acquisition in P300 speller systems. Clin Neurophysiol 2014; 126:1171-1177. [PMID: 25316166 DOI: 10.1016/j.clinph.2014.09.021] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2014] [Revised: 09/04/2014] [Accepted: 09/22/2014] [Indexed: 10/24/2022]
Abstract
OBJECTIVE The P300 speller is intended to restore communication to patients with advanced neuromuscular disorders, but clinical implementation may be hindered by several factors, including system setup, burden, and cost. Our goal was to develop a method that can overcome these barriers by optimizing EEG electrode number and placement for P300 studies within a population of subjects. METHODS A Gibbs sampling method was developed to find the optimal electrode configuration given a set of P300 speller data. The method was tested on a set of data from 15 healthy subjects using an established 32-electrode pattern. Resulting electrode configurations were then validated using online prospective testing with a naïve Bayes classifier in 15 additional healthy subjects. RESULTS The method yielded a set of four posterior electrodes (PO₈, PO₇, POZ, CPZ), which produced results that are likely sufficient to be clinically effective. In online prospective validation testing, no significant difference was found between subjects' performances using the reduced and the full electrode configurations. CONCLUSIONS The proposed method can find reduced sets of electrodes within a subject population without reducing performance. SIGNIFICANCE Reducing the number of channels may reduce costs, set-up time, signal bandwidth, and computation requirements for practical online P300 speller implementation.
Collapse
Affiliation(s)
- William Speier
- University of California, Los Angeles, Department of Bioengineering, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA; University of California, Los Angeles, Medical Imaging Informatics Group, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA
| | - Aniket Deshpande
- University of California, Los Angeles, Department of Bioengineering, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA
| | - Nader Pouratian
- University of California, Los Angeles, Department of Bioengineering, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA; University of California, Los Angeles, Department of Neurosurgery, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA; University of California, Los Angeles, Interdepartmental Program in Neuroscience, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA; University of California, Los Angeles, Brain Research Institute, David Geffen School of Medicine at UCLA, Los Angeles, CA, USA.
| |
Collapse
|
32
|
Cecotti H, Rivet B. Correction: cecotti, h. And rivet, B. Subject combination and electrode selection in cooperative brain-computer interface based on event related potentials. Brain sci. 2014, 4, 335-355. Brain Sci 2014; 4:488-508. [PMID: 25243772 PMCID: PMC4194035 DOI: 10.3390/brainsci4030488] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2014] [Accepted: 09/09/2014] [Indexed: 11/16/2022] Open
Abstract
The authors wish to make the following correction to this paper (Cecotti, H.; Rivet, B. Subject Combination and Electrode Selection in Cooperative Brain-Computer Interface Based on Event Related Potentials. Brain Sci. 2014, 4, 335-355). Dut to an error the reference number in the original published paper were not shown. The former main text should be replaced as below.
Collapse
Affiliation(s)
- Hubert Cecotti
- School of Computing and Intelligent Systems, University of Ulster, Derry BT48 7JL, Northern Ireland, UK.
| | - Bertrand Rivet
- GIPSA-lab CNRS UMR 5216, Grenoble Universities, Saint Martin d'Hères 38400, France.
| |
Collapse
|
33
|
Colwell K, Throckmorton C, Collins L, Morton K. Projected Accuracy Metric for the P300 Speller. IEEE Trans Neural Syst Rehabil Eng 2014; 22:921-5. [DOI: 10.1109/tnsre.2014.2324892] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
34
|
Kindermans PJ, Tangermann M, Müller KR, Schrauwen B. Integrating dynamic stopping, transfer learning and language models in an adaptive zero-training ERP speller. J Neural Eng 2014; 11:035005. [PMID: 24834896 DOI: 10.1088/1741-2560/11/3/035005] [Citation(s) in RCA: 65] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Most BCIs have to undergo a calibration session in which data is recorded to train decoders with machine learning. Only recently zero-training methods have become a subject of study. This work proposes a probabilistic framework for BCI applications which exploit event-related potentials (ERPs). For the example of a visual P300 speller we show how the framework harvests the structure suitable to solve the decoding task by (a) transfer learning, (b) unsupervised adaptation, (c) language model and (d) dynamic stopping. APPROACH A simulation study compares the proposed probabilistic zero framework (using transfer learning and task structure) to a state-of-the-art supervised model on n = 22 subjects. The individual influence of the involved components (a)-(d) are investigated. MAIN RESULTS Without any need for a calibration session, the probabilistic zero-training framework with inter-subject transfer learning shows excellent performance--competitive to a state-of-the-art supervised method using calibration. Its decoding quality is carried mainly by the effect of transfer learning in combination with continuous unsupervised adaptation. SIGNIFICANCE A high-performing zero-training BCI is within reach for one of the most popular BCI paradigms: ERP spelling. Recording calibration data for a supervised BCI would require valuable time which is lost for spelling. The time spent on calibration would allow a novel user to spell 29 symbols with our unsupervised approach. It could be of use for various clinical and non-clinical ERP-applications of BCI.
Collapse
Affiliation(s)
- Pieter-Jan Kindermans
- Electronics and Information Systems (ELIS) Department, Ghent University, Sint Pietersnieuwstraat 41, B-9000 Ghent, Belgium
| | | | | | | |
Collapse
|
35
|
Thompson DE, Quitadamo LR, Mainardi L, Laghari KUR, Gao S, Kindermans PJ, Simeral JD, Fazel-Rezai R, Matteucci M, Falk TH, Bianchi L, Chestek CA, Huggins JE. Performance measurement for brain-computer or brain-machine interfaces: a tutorial. J Neural Eng 2014; 11:035001. [PMID: 24838070 DOI: 10.1088/1741-2560/11/3/035001] [Citation(s) in RCA: 52] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
OBJECTIVE Brain-computer interfaces (BCIs) have the potential to be valuable clinical tools. However, the varied nature of BCIs, combined with the large number of laboratories participating in BCI research, makes uniform performance reporting difficult. To address this situation, we present a tutorial on performance measurement in BCI research. APPROACH A workshop on this topic was held at the 2013 International BCI Meeting at Asilomar Conference Center in Pacific Grove, California. This paper contains the consensus opinion of the workshop members, refined through discussion in the following months and the input of authors who were unable to attend the workshop. MAIN RESULTS Checklists for methods reporting were developed for both discrete and continuous BCIs. Relevant metrics are reviewed for different types of BCI research, with notes on their use to encourage uniform application between laboratories. SIGNIFICANCE Graduate students and other researchers new to BCI research may find this tutorial a helpful introduction to performance measurement in the field.
Collapse
Affiliation(s)
- David E Thompson
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA
| | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
36
|
Mainsah BO, Colwell KA, Collins LM, Throckmorton CS. Utilizing a language model to improve online dynamic data collection in P300 spellers. IEEE Trans Neural Syst Rehabil Eng 2014; 22:837-46. [PMID: 24808413 DOI: 10.1109/tnsre.2014.2321290] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
P300 spellers provide a means of communication for individuals with severe physical limitations, especially those with locked-in syndrome, such as amyotrophic lateral sclerosis. However, P300 speller use is still limited by relatively low communication rates due to the multiple data measurements that are required to improve the signal-to-noise ratio of event-related potentials for increased accuracy. Therefore, the amount of data collection has competing effects on accuracy and spelling speed. Adaptively varying the amount of data collection prior to character selection has been shown to improve spelling accuracy and speed. The goal of this study was to optimize a previously developed dynamic stopping algorithm that uses a Bayesian approach to control data collection by incorporating a priori knowledge via a language model. Participants ( n = 17) completed online spelling tasks using the dynamic stopping algorithm, with and without a language model. The addition of the language model resulted in improved participant performance from a mean theoretical bit rate of 46.12 bits/min at 88.89% accuracy to 54.42 bits/min ( ) at 90.36% accuracy.
Collapse
|
37
|
Cecotti H, Rivet B. Subject combination and electrode selection in cooperative brain-computer interface based on event related potentials. Brain Sci 2014; 4:335-55. [PMID: 24961765 PMCID: PMC4101481 DOI: 10.3390/brainsci4020335] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2014] [Revised: 03/18/2014] [Accepted: 03/20/2014] [Indexed: 11/16/2022] Open
Abstract
New paradigms are required in Brain-Computer Interface (BCI) systems for the needs and expectations of healthy people. To solve this issue, we explore the emerging field of cooperative BCIs, which involves several users in a single BCI system. Contrary to classical BCIs that are dependent on the unique subject's will, cooperative BCIs are used for problem solving tasks where several people shall be engaged by sharing a common goal. Similarly as combining trials over time improves performance, combining trials across subjects can significantly improve performance compared with when only a single user is involved. Yet, cooperative BCIs may only be used in particular settings, and new paradigms must be proposed to efficiently use this approach. The possible benefits of using several subjects are addressed, and compared with current single-subject BCI paradigms. To show the advantages of a cooperative BCI, we evaluate the performance of combining decisions across subjects with data from an event-related potentials (ERP) based experiment where each subject observed the same sequence of visual stimuli. Furthermore, we show that it is possible to achieve a mean AUC superior to 0.95 with 10 subjects and 3 electrodes on each subject, or with 4 subjects and 6 electrodes on each subject. Several emerging challenges and possible applications are proposed to highlight how cooperative BCIs could be efficiently used with current technologies and leverage BCI applications.
Collapse
Affiliation(s)
- Hubert Cecotti
- School of Computing and Intelligent Systems, University of Ulster, Derry BT48 7JL, Northern Ireland, UK.
| | - Bertrand Rivet
- GIPSA-lab CNRS UMR 5216, Grenoble Universities, Saint Martin d'Hères 38400, France.
| |
Collapse
|
38
|
Mora-Cortes A, Manyakov NV, Chumerin N, Van Hulle MM. Language model applications to spelling with Brain-Computer Interfaces. SENSORS (BASEL, SWITZERLAND) 2014; 14:5967-93. [PMID: 24675760 PMCID: PMC4029701 DOI: 10.3390/s140405967] [Citation(s) in RCA: 28] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/23/2013] [Revised: 02/17/2014] [Accepted: 02/24/2014] [Indexed: 11/16/2022]
Abstract
Within the Ambient Assisted Living (AAL) community, Brain-Computer Interfaces (BCIs) have raised great hopes as they provide alternative communication means for persons with disabilities bypassing the need for speech and other motor activities. Although significant advancements have been realized in the last decade, applications of language models (e.g., word prediction, completion) have only recently started to appear in BCI systems. The main goal of this article is to review the language model applications that supplement non-invasive BCI-based communication systems by discussing their potential and limitations, and to discern future trends. First, a brief overview of the most prominent BCI spelling systems is given, followed by an in-depth discussion of the language models applied to them. These language models are classified according to their functionality in the context of BCI-based spelling: the static/dynamic nature of the user interface, the use of error correction and predictive spelling, and the potential to improve their classification performance by using language models. To conclude, the review offers an overview of the advantages and challenges when implementing language models in BCI-based communication systems when implemented in conjunction with other AAL technologies.
Collapse
Affiliation(s)
- Anderson Mora-Cortes
- Laboratorium voor Neuro- en Psychofysiologie, KU Leuven, Campus Gasthuisberg, O&N2, Herestraat 49, Leuven B-3000, Belgium.
| | - Nikolay V Manyakov
- Laboratorium voor Neuro- en Psychofysiologie, KU Leuven, Campus Gasthuisberg, O&N2, Herestraat 49, Leuven B-3000, Belgium.
| | - Nikolay Chumerin
- Laboratorium voor Neuro- en Psychofysiologie, KU Leuven, Campus Gasthuisberg, O&N2, Herestraat 49, Leuven B-3000, Belgium.
| | - Marc M Van Hulle
- Laboratorium voor Neuro- en Psychofysiologie, KU Leuven, Campus Gasthuisberg, O&N2, Herestraat 49, Leuven B-3000, Belgium.
| |
Collapse
|
39
|
Speier W, Arnold C, Lu J, Deshpande A, Pouratian N. Integrating language information with a hidden Markov model to improve communication rate in the P300 speller. IEEE Trans Neural Syst Rehabil Eng 2014; 22:678-84. [PMID: 24760927 DOI: 10.1109/tnsre.2014.2300091] [Citation(s) in RCA: 30] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
The P300 speller is a common brain-computer interface (BCI) application designed to communicate language by detecting event related potentials in a subject's electroencephalogram (EEG) signal. Information about the structure of natural language can be valuable for BCI communication systems, but few attempts have been made to incorporate this domain knowledge into the classifier. In this study, we treat BCI communication as a hidden Markov model (HMM) where hidden states are target characters and the EEG signal is the visible output. Using the Viterbi algorithm, language information can be incorporated in classification and errors can be corrected automatically. This method was first evaluated offline on a dataset of 15 healthy subjects who had a significant increase in bit rate from a previously published naïve Bayes approach and an average 32% increase from standard classification with dynamic stopping. An online pilot study of five healthy subjects verified these results as the average bit rate achieved using the HMM method was significantly higher than that using the naïve Bayes and standard methods. These findings strongly support the integration of domain-specific knowledge into BCI classification to improve system performance and accuracy.
Collapse
|
40
|
Akcakaya M, Peters B, Moghadamfalahi M, Mooney AR, Orhan U, Oken B, Erdogmus D, Fried-Oken M. Noninvasive brain-computer interfaces for augmentative and alternative communication. IEEE Rev Biomed Eng 2014; 7:31-49. [PMID: 24802700 PMCID: PMC6525622 DOI: 10.1109/rbme.2013.2295097] [Citation(s) in RCA: 66] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/08/2023]
Abstract
Brain-computer interfaces (BCIs) promise to provide a novel access channel for assistive technologies, including augmentative and alternative communication (AAC) systems, to people with severe speech and physical impairments (SSPI). Research on the subject has been accelerating significantly in the last decade and the research community took great strides toward making BCI-AAC a practical reality to individuals with SSPI. Nevertheless, the end goal has still not been reached and there is much work to be done to produce real-world-worthy systems that can be comfortably, conveniently, and reliably used by individuals with SSPI with help from their families and care givers who will need to maintain, setup, and debug the systems at home. This paper reviews reports in the BCI field that aim at AAC as the application domain with a consideration on both technical and clinical aspects.
Collapse
|
41
|
Hill K, Kovacs T, Shin S. Reliability of brain computer interface language sample transcription procedures. ACTA ACUST UNITED AC 2014; 51:579-90. [DOI: 10.1682/jrrd.2013.05.0102] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
42
|
Oken BS, Orhan U, Roark B, Erdogmus D, Fowler A, Mooney A, Peters B, Miller M, Fried-Oken MB. Brain-computer interface with language model-electroencephalography fusion for locked-in syndrome. Neurorehabil Neural Repair 2013; 28:387-94. [PMID: 24370570 DOI: 10.1177/1545968313516867] [Citation(s) in RCA: 50] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
BACKGROUND Some noninvasive brain-computer interface (BCI) systems are currently available for locked-in syndrome (LIS) but none have incorporated a statistical language model during text generation. OBJECTIVE To begin to address the communication needs of individuals with LIS using a noninvasive BCI that involves rapid serial visual presentation (RSVP) of symbols and a unique classifier with electroencephalography (EEG) and language model fusion. METHODS The RSVP Keyboard was developed with several unique features. Individual letters are presented at 2.5 per second. Computer classification of letters as targets or nontargets based on EEG is performed using machine learning that incorporates a language model for letter prediction via Bayesian fusion enabling targets to be presented only 1 to 4 times. Nine participants with LIS and 9 healthy controls were enrolled. After screening, subjects first calibrated the system, and then completed a series of balanced word generation mastery tasks that were designed with 5 incremental levels of difficulty, which increased by selecting phrases for which the utility of the language model decreased naturally. RESULTS Six participants with LIS and 9 controls completed the experiment. All LIS participants successfully mastered spelling at level 1 and one subject achieved level 5. Six of 9 control participants achieved level 5. CONCLUSIONS Individuals who have incomplete LIS may benefit from an EEG-based BCI system, which relies on EEG classification and a statistical language model. Steps to further improve the system are discussed.
Collapse
Affiliation(s)
- Barry S Oken
- 1Oregon Health & Science University, Portland, OR, USA
| | | | | | | | | | | | | | | | | |
Collapse
|
43
|
Akram F, Han HS, Kim TS. A P300-based brain computer interface system for words typing. Comput Biol Med 2013; 45:118-25. [PMID: 24480171 DOI: 10.1016/j.compbiomed.2013.12.001] [Citation(s) in RCA: 18] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2013] [Revised: 11/29/2013] [Accepted: 12/03/2013] [Indexed: 10/25/2022]
Abstract
P300 is an event related potential of the brain in response to oddball events. Brain Computer Interface (BCI) utilizing P300 is known as a P300 BCI system. A conventional P300 BCI system for character spelling is composed of a paradigm that displays flashing characters and a classification scheme which identifies target characters. To type a word a user has to spell each character of the word: this spelling process is slow and it can take several minutes to type a word. In this study, we propose a new word typing scheme by integrating a word suggestion mechanism with a dictionary search into the conventional P300-based speller. Our new P300-based word typing system consists of an initial character spelling paradigm, a dictionary unit to give suggestions of possible words and the second word selection paradigm to select a word out of the suggestions. Our proposed methodology reduces typing time significantly and makes word typing easy via a P300 BCI system. We have tested our system with ten subjects and our results demonstrate an average word typing time of 1.91 min whereas the conventional took 3.36 min for the same words.
Collapse
Affiliation(s)
- Faraz Akram
- Department of Biomedical Engineering, Kyung Hee University, Republic of Korea
| | - Hee-Sok Han
- Department of Biomedical Engineering, Kyung Hee University, Republic of Korea
| | - Tae-Seong Kim
- Department of Biomedical Engineering, Kyung Hee University, Republic of Korea.
| |
Collapse
|
44
|
Speier W, Arnold C, Pouratian N. Evaluating true BCI communication rate through mutual information and language models. PLoS One 2013; 8:e78432. [PMID: 24167623 PMCID: PMC3805537 DOI: 10.1371/journal.pone.0078432] [Citation(s) in RCA: 26] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2013] [Accepted: 09/10/2013] [Indexed: 11/19/2022] Open
Abstract
Brain-computer interface (BCI) systems are a promising means for restoring communication to patients suffering from “locked-in” syndrome. Research to improve system performance primarily focuses on means to overcome the low signal to noise ratio of electroencephalogric (EEG) recordings. However, the literature and methods are difficult to compare due to the array of evaluation metrics and assumptions underlying them, including that: 1) all characters are equally probable, 2) character selection is memoryless, and 3) errors occur completely at random. The standardization of evaluation metrics that more accurately reflect the amount of information contained in BCI language output is critical to make progress. We present a mutual information-based metric that incorporates prior information and a model of systematic errors. The parameters of a system used in one study were re-optimized, showing that the metric used in optimization significantly affects the parameter values chosen and the resulting system performance. The results of 11 BCI communication studies were then evaluated using different metrics, including those previously used in BCI literature and the newly advocated metric. Six studies' results varied based on the metric used for evaluation and the proposed metric produced results that differed from those originally published in two of the studies. Standardizing metrics to accurately reflect the rate of information transmission is critical to properly evaluate and compare BCI communication systems and advance the field in an unbiased manner.
Collapse
Affiliation(s)
- William Speier
- Department of Bioengineering, University of California Los Angeles, Los Angeles, California, United States of America
- Medical Imaging Informatics Group, University of California Los Angeles, Los Angeles, California, United States of America
| | - Corey Arnold
- Medical Imaging Informatics Group, University of California Los Angeles, Los Angeles, California, United States of America
| | - Nader Pouratian
- Department of Bioengineering, University of California Los Angeles, Los Angeles, California, United States of America
- Department of Neurosurgery, University of California Los Angeles, Los Angeles, California, United States of America
- Interdepartmental Program in Neuroscience, University of California Los Angeles, Los Angeles, California, United States of America
- Brain Research Institute, University of California Los Angeles, Los Angeles, California, United States of America
- * E-mail:
| |
Collapse
|
45
|
Orhan U, Erdogmus D, Roark B, Oken B, Fried-Oken M. Offline analysis of context contribution to ERP-based typing BCI performance. J Neural Eng 2013; 10:066003. [PMID: 24099944 DOI: 10.1088/1741-2560/10/6/066003] [Citation(s) in RCA: 27] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE We aim to increase the symbol rate of electroencephalography (EEG) based brain-computer interface (BCI) typing systems by utilizing context information. APPROACH Event related potentials (ERP) corresponding to a stimulus in EEG can be used to detect the intended target of a person for BCI. This paradigm is widely utilized to build letter-by-letter BCI typing systems. Nevertheless currently available BCI typing systems still require improvement due to low typing speeds. This is mainly due to the reliance on multiple repetitions before making a decision to achieve higher typing accuracy. Another possible approach to increase the speed of typing while not significantly reducing the accuracy of typing is to use additional context information. In this paper, we study the effect of using a language model (LM) as additional evidence for intent detection. Bayesian fusion of an n-gram symbol model with EEG features is proposed, and a specifically regularized discriminant analysis ERP discriminant is used to obtain EEG-based features. The target detection accuracies are rigorously evaluated for varying LM orders, as well as the number of ERP-inducing repetitions. MAIN RESULTS The results demonstrate that the LMs contribute significantly to letter classification accuracy. For instance, we find that a single-trial ERP detection supported by a 4-gram LM may achieve the same performance as using 3-trial ERP classification for the non-initial letters of words. SIGNIFICANCE Overall, the fusion of evidence from EEG and LMs yields a significant opportunity to increase the symbol rate of a BCI typing system.
Collapse
Affiliation(s)
- Umut Orhan
- Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA
| | | | | | | | | |
Collapse
|
46
|
Yaming Xu, Nakajima Y. A Two-Level Predictive Event-Related Potential-Based Brain–Computer Interface. IEEE Trans Biomed Eng 2013; 60:2839-47. [DOI: 10.1109/tbme.2013.2265103] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
47
|
Kaufmann T, Holz EM, Kübler A. Comparison of tactile, auditory, and visual modality for brain-computer interface use: a case study with a patient in the locked-in state. Front Neurosci 2013; 7:129. [PMID: 23898236 PMCID: PMC3721006 DOI: 10.3389/fnins.2013.00129] [Citation(s) in RCA: 99] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2013] [Accepted: 07/05/2013] [Indexed: 12/03/2022] Open
Abstract
This paper describes a case study with a patient in the classic locked-in state, who currently has no means of independent communication. Following a user-centered approach, we investigated event-related potentials (ERP) elicited in different modalities for use in brain-computer interface (BCI) systems. Such systems could provide her with an alternative communication channel. To investigate the most viable modality for achieving BCI based communication, classic oddball paradigms (1 rare and 1 frequent stimulus, ratio 1:5) in the visual, auditory and tactile modality were conducted (2 runs per modality). Classifiers were built on one run and tested offline on another run (and vice versa). In these paradigms, the tactile modality was clearly superior to other modalities, displaying high offline accuracy even when classification was performed on single trials only. Consequently, we tested the tactile paradigm online and the patient successfully selected targets without any error. Furthermore, we investigated use of the visual or tactile modality for different BCI systems with more than two selection options. In the visual modality, several BCI paradigms were tested offline. Neither matrix-based nor so-called gaze-independent paradigms constituted a means of control. These results may thus question the gaze-independence of current gaze-independent approaches to BCI. A tactile four-choice BCI resulted in high offline classification accuracies. Yet, online use raised various issues. Although performance was clearly above chance, practical daily life use appeared unlikely when compared to other communication approaches (e.g., partner scanning). Our results emphasize the need for user-centered design in BCI development including identification of the best stimulus modality for a particular user. Finally, the paper discusses feasibility of EEG-based BCI systems for patients in classic locked-in state and compares BCI to other AT solutions that we also tested during the study.
Collapse
Affiliation(s)
- Tobias Kaufmann
- Department for Psychology I, Institute for Psychology, University of Würzburg Würzburg, Germany
| | | | | |
Collapse
|
48
|
Cognitive-motor brain-machine interfaces. ACTA ACUST UNITED AC 2013; 108:38-44. [PMID: 23774120 DOI: 10.1016/j.jphysparis.2013.05.005] [Citation(s) in RCA: 24] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2012] [Revised: 03/27/2013] [Accepted: 05/23/2013] [Indexed: 11/21/2022]
Abstract
Brain-machine interfaces (BMIs) open new horizons for the treatment of paralyzed persons, giving hope for the artificial restoration of lost physiological functions. Whereas BMI development has mainly focused on motor rehabilitation, recent studies have suggested that higher cognitive functions can also be deciphered from brain activity, bypassing low level planning and execution functions, and replacing them by computer-controlled effectors. This review describes the new generation of cognitive-motor BMIs, focusing on three BMI types: By outlining recent progress in developing these BMI types, we aim to provide a unified view of contemporary research towards the replacement of behavioral outputs of cognitive processes by direct interaction with the brain.
Collapse
|
49
|
Thompson DE, Blain-Moraes S, Huggins JE. Performance assessment in brain-computer interface-based augmentative and alternative communication. Biomed Eng Online 2013; 12:43. [PMID: 23680020 PMCID: PMC3662584 DOI: 10.1186/1475-925x-12-43] [Citation(s) in RCA: 38] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2012] [Accepted: 04/17/2013] [Indexed: 11/14/2022] Open
Abstract
A large number of incommensurable metrics are currently used to report the performance of brain-computer interfaces (BCI) used for augmentative and alterative communication (AAC). The lack of standard metrics precludes the comparison of different BCI-based AAC systems, hindering rapid growth and development of this technology. This paper presents a review of the metrics that have been used to report performance of BCIs used for AAC from January 2005 to January 2012. We distinguish between Level 1 metrics used to report performance at the output of the BCI Control Module, which translates brain signals into logical control output, and Level 2 metrics at the Selection Enhancement Module, which translates logical control to semantic control. We recommend that: (1) the commensurate metrics Mutual Information or Information Transfer Rate (ITR) be used to report Level 1 BCI performance, as these metrics represent information throughput, which is of interest in BCIs for AAC; 2) the BCI-Utility metric be used to report Level 2 BCI performance, as it is capable of handling all current methods of improving BCI performance; (3) these metrics should be supplemented by information specific to each unique BCI configuration; and (4) studies involving Selection Enhancement Modules should report performance at both Level 1 and Level 2 in the BCI system. Following these recommendations will enable efficient comparison between both BCI Control and Selection Enhancement Modules, accelerating research and development of BCI-based AAC systems.
Collapse
Affiliation(s)
- David E Thompson
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA
| | - Stefanie Blain-Moraes
- Department of Physical Medicine and Rehabilitation, University of Michigan, Ann Arbor, MI, USA
| | - Jane E Huggins
- Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA
- Department of Physical Medicine and Rehabilitation, University of Michigan, Ann Arbor, MI, USA
| |
Collapse
|
50
|
Kindermans PJ, Verschore H, Schrauwen B. A unified probabilistic approach to improve spelling in an event-related potential-based brain-computer interface. IEEE Trans Biomed Eng 2013; 60:2696-705. [PMID: 23674419 DOI: 10.1109/tbme.2013.2262524] [Citation(s) in RCA: 28] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
In recent years, in an attempt to maximize performance, machine learning approaches for event-related potential (ERP) spelling have become more and more complex. In this paper, we have taken a step back as we wanted to improve the performance without building an overly complex model, that cannot be used by the community. Our research resulted in a unified probabilistic model for ERP spelling, which is based on only three assumptions and incorporates language information. On top of that, the probabilistic nature of our classifier yields a natural dynamic stopping strategy. Furthermore, our method uses the same parameters across 25 subjects from three different datasets. We show that our classifier, when enhanced with language models and dynamic stopping, improves the spelling speed and accuracy drastically. Additionally, we would like to point out that as our model is entirely probabilistic, it can easily be used as the foundation for complex systems in future work. All our experiments are executed on publicly available datasets to allow for future comparison with similar techniques.
Collapse
|