1
|
Liu X, Hu B, Si Y, Wang Q. The role of eye movement signals in non-invasive brain-computer interface typing system. Med Biol Eng Comput 2024; 62:1981-1990. [PMID: 38509350 DOI: 10.1007/s11517-024-03070-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Accepted: 03/05/2024] [Indexed: 03/22/2024]
Abstract
Brain-Computer Interfaces (BCIs) have shown great potential in providing communication and control for individuals with severe motor disabilities. However, traditional BCIs that rely on electroencephalography (EEG) signals suffer from low information transfer rates and high variability across users. Recently, eye movement signals have emerged as a promising alternative due to their high accuracy and robustness. Eye movement signals are the electrical or mechanical signals generated by the movements and behaviors of the eyes, serving to denote the diverse forms of eye movements, such as fixations, smooth pursuit, and other oculomotor activities like blinking. This article presents a review of recent studies on the development of BCI typing systems that incorporate eye movement signals. We first discuss the basic principles of BCI and the recent advancements in text entry. Then, we provide a comprehensive summary of the latest advancements in BCI typing systems that leverage eye movement signals. This includes an in-depth analysis of hybrid BCIs that are built upon the integration of electrooculography (EOG) and eye tracking technology, aiming to enhance the performance and functionality of the system. Moreover, we highlight the advantages and limitations of different approaches, as well as potential future directions. Overall, eye movement signals hold great potential for enhancing the usability and accessibility of BCI typing systems, and further research in this area could lead to more effective communication and control for individuals with motor disabilities.
Collapse
Affiliation(s)
- Xi Liu
- Key Laboratory of Spectral Imaging Technology, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, 710119, China
- University of Chinese Academy of Sciences, Beijing, 100049, China
- Key Laboratory of Biomedical Spectroscopy of Xi'an, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, 710119, China
| | - Bingliang Hu
- Key Laboratory of Spectral Imaging Technology, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, 710119, China
- Key Laboratory of Biomedical Spectroscopy of Xi'an, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, 710119, China
| | - Yang Si
- Department of Neurology, Sichuan Academy of Medical Science and Sichuan Provincial People's Hospital, Chengdu, 611731, China
- University of Electronic Science and Technology of China, Chengdu, 611731, China
| | - Quan Wang
- Key Laboratory of Spectral Imaging Technology, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, 710119, China.
- Key Laboratory of Biomedical Spectroscopy of Xi'an, Xi'an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences, Xi'an, 710119, China.
| |
Collapse
|
2
|
Wang R, Chen ZS. Large-scale foundation models and generative AI for BigData neuroscience. Neurosci Res 2024:S0168-0102(24)00075-0. [PMID: 38897235 DOI: 10.1016/j.neures.2024.06.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2023] [Revised: 04/15/2024] [Accepted: 05/15/2024] [Indexed: 06/21/2024]
Abstract
Recent advances in machine learning have led to revolutionary breakthroughs in computer games, image and natural language understanding, and scientific discovery. Foundation models and large-scale language models (LLMs) have recently achieved human-like intelligence thanks to BigData. With the help of self-supervised learning (SSL) and transfer learning, these models may potentially reshape the landscapes of neuroscience research and make a significant impact on the future. Here we present a mini-review on recent advances in foundation models and generative AI models as well as their applications in neuroscience, including natural language and speech, semantic memory, brain-machine interfaces (BMIs), and data augmentation. We argue that this paradigm-shift framework will open new avenues for many neuroscience research directions and discuss the accompanying challenges and opportunities.
Collapse
Affiliation(s)
- Ran Wang
- Department of Psychiatry, New York University Grossman School of Medicine, New York, NY 10016, USA
| | - Zhe Sage Chen
- Department of Psychiatry, New York University Grossman School of Medicine, New York, NY 10016, USA; Department of Neuroscience and Physiology, Neuroscience Institute, New York University Grossman School of Medicine, New York, NY 10016, USA; Department of Biomedical Engineering, New York University Tandon School of Engineering, Brooklyn, NY 11201, USA.
| |
Collapse
|
3
|
Kosnoff J, Yu K, Liu C, He B. Transcranial focused ultrasound to V5 enhances human visual motion brain-computer interface by modulating feature-based attention. Nat Commun 2024; 15:4382. [PMID: 38862476 PMCID: PMC11167030 DOI: 10.1038/s41467-024-48576-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Accepted: 05/02/2024] [Indexed: 06/13/2024] Open
Abstract
A brain-computer interface (BCI) enables users to control devices with their minds. Despite advancements, non-invasive BCIs still exhibit high error rates, prompting investigation into the potential reduction through concurrent targeted neuromodulation. Transcranial focused ultrasound (tFUS) is an emerging non-invasive neuromodulation technology with high spatiotemporal precision. This study examines whether tFUS neuromodulation can improve BCI outcomes, and explores the underlying mechanism of action using high-density electroencephalography (EEG) source imaging (ESI). As a result, V5-targeted tFUS significantly reduced the error in a BCI speller task. Source analyses revealed a significantly increase in theta and alpha activities in the tFUS condition at both V5 and downstream in the dorsal visual processing pathway. Correlation analysis indicated that the connection within the dorsal processing pathway was preserved during tFUS stimulation, while the ventral connection was weakened. These findings suggest that V5-targeted tFUS enhances feature-based attention to visual motion.
Collapse
Affiliation(s)
- Joshua Kosnoff
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15237, USA
| | - Kai Yu
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15237, USA
| | - Chang Liu
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15237, USA
- Department of Biomedical Engineering, Boston University, Boston, MA, 02215, USA
| | - Bin He
- Department of Biomedical Engineering, Carnegie Mellon University, Pittsburgh, PA, 15237, USA.
- Neuroscience Institute, Carnegie Mellon University, Pittsburgh, PA, 15237, USA.
| |
Collapse
|
4
|
Eldawlatly S. On the role of generative artificial intelligence in the development of brain-computer interfaces. BMC Biomed Eng 2024; 6:4. [PMID: 38698495 PMCID: PMC11064240 DOI: 10.1186/s42490-024-00080-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2023] [Accepted: 04/24/2024] [Indexed: 05/05/2024] Open
Abstract
Since their inception more than 50 years ago, Brain-Computer Interfaces (BCIs) have held promise to compensate for functions lost by people with disabilities through allowing direct communication between the brain and external devices. While research throughout the past decades has demonstrated the feasibility of BCI to act as a successful assistive technology, the widespread use of BCI outside the lab is still beyond reach. This can be attributed to a number of challenges that need to be addressed for BCI to be of practical use including limited data availability, limited temporal and spatial resolutions of brain signals recorded non-invasively and inter-subject variability. In addition, for a very long time, BCI development has been mainly confined to specific simple brain patterns, while developing other BCI applications relying on complex brain patterns has been proven infeasible. Generative Artificial Intelligence (GAI) has recently emerged as an artificial intelligence domain in which trained models can be used to generate new data with properties resembling that of available data. Given the enhancements observed in other domains that possess similar challenges to BCI development, GAI has been recently employed in a multitude of BCI development applications to generate synthetic brain activity; thereby, augmenting the recorded brain activity. Here, a brief review of the recent adoption of GAI techniques to overcome the aforementioned BCI challenges is provided demonstrating the enhancements achieved using GAI techniques in augmenting limited EEG data, enhancing the spatiotemporal resolution of recorded EEG data, enhancing cross-subject performance of BCI systems and implementing end-to-end BCI applications. GAI could represent the means by which BCI would be transformed into a prevalent assistive technology, thereby improving the quality of life of people with disabilities, and helping in adopting BCI as an emerging human-computer interaction technology for general use.
Collapse
Affiliation(s)
- Seif Eldawlatly
- Computer and Systems Engineering Department, Faculty of Engineering, Ain Shams University, 1 El-Sarayat St., Abbassia, Cairo, Egypt.
- Computer Science and Engineering Department, The American University in Cairo, Cairo, Egypt.
| |
Collapse
|
5
|
Herbert C. Brain-computer interfaces and human factors: the role of language and cultural differences-Still a missing gap? Front Hum Neurosci 2024; 18:1305445. [PMID: 38665897 PMCID: PMC11043545 DOI: 10.3389/fnhum.2024.1305445] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2023] [Accepted: 02/02/2024] [Indexed: 04/28/2024] Open
Abstract
Brain-computer interfaces (BCIs) aim at the non-invasive investigation of brain activity for supporting communication and interaction of the users with their environment by means of brain-machine assisted technologies. Despite technological progress and promising research aimed at understanding the influence of human factors on BCI effectiveness, some topics still remain unexplored. The aim of this article is to discuss why it is important to consider the language of the user, its embodied grounding in perception, action and emotions, and its interaction with cultural differences in information processing in future BCI research. Based on evidence from recent studies, it is proposed that detection of language abilities and language training are two main topics of enquiry of future BCI studies to extend communication among vulnerable and healthy BCI users from bench to bedside and real world applications. In addition, cultural differences shape perception, actions, cognition, language and emotions subjectively, behaviorally as well as neuronally. Therefore, BCI applications should consider cultural differences in information processing to develop culture- and language-sensitive BCI applications for different user groups and BCIs, and investigate the linguistic and cultural contexts in which the BCI will be used.
Collapse
Affiliation(s)
- Cornelia Herbert
- Applied Emotion and Motivation Psychology, Institute of Psychology and Education, Ulm University, Ulm, Germany
| |
Collapse
|
6
|
Klein E, Kinsella M, Stevens I, Fried-Oken M. Ethical issues raised by incorporating personalized language models into brain-computer interface communication technologies: a qualitative study of individuals with neurological disease. Disabil Rehabil Assist Technol 2024; 19:1041-1051. [PMID: 36403143 PMCID: PMC10351684 DOI: 10.1080/17483107.2022.2146217] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2022] [Revised: 09/01/2022] [Accepted: 11/07/2022] [Indexed: 11/21/2022]
Abstract
PURPOSE To examine the views of individuals with neurodegenerative diseases about ethical issues related to incorporating personalized language models into brain-computer interface (BCI) communication technologies. METHODS Fifteen semi-structured interviews and 51 online free response surveys were completed with individuals diagnosed with neurodegenerative disease that could lead to loss of speech and motor skills. Each participant responded to questions after six hypothetical ethics vignettes were presented that address the possibility of building language models with personal words and phrases in BCI communication technologies. Data were analyzed with consensus coding, using modified grounded theory. RESULTS Four themes were identified. (1) The experience of a neurodegenerative disease shapes preferences for personalized language models. (2) An individual's identity will be affected by the ability to personalize the language model. (3) The motivation for personalization is tied to how relationships can be helped or harmed. (4) Privacy is important to people who may need BCI communication technologies. Responses suggest that the inclusion of personal lexica raises ethical issues. Stakeholders want their values to be considered during development of BCI communication technologies. CONCLUSIONS With the rapid development of BCI communication technologies, it is critical to incorporate feedback from individuals regarding their ethical concerns about the storage and use of personalized language models. Stakeholder values and preferences about disability, privacy, identity and relationships should drive design, innovation and implementation.IMPLICATIONS FOR REHABILITATIONIndividuals with neurodegenerative diseases are important stakeholders to consider in development of natural language processing within brain-computer interface (BCI) communication technologies.The incorporation of personalized language models raises issues related to disability, identity, relationships, and privacy.People who may one day rely on BCI communication technologies care not just about usability of communication technology but about technology that supports their values and priorities.Qualitative ethics-focused research is a valuable tool for exploring stakeholder perspectives on new capabilities of BCI communication technologies, such as the storage and use of personalized language models.
Collapse
Affiliation(s)
- Eran Klein
- Department of Neurology, Oregon Health & Science University, Portland, OR USA
| | - Michelle Kinsella
- Institute on Development and Disability, Oregon Health & Science University, Portland, OR USA
| | - Ian Stevens
- Department of Neurosurgery, Oregon Health & Science University, Portland, OR USA
| | - Melanie Fried-Oken
- Department of Neurology, Oregon Health & Science University, Portland, OR USA
- Institute on Development and Disability, Oregon Health & Science University, Portland, OR USA
| |
Collapse
|
7
|
Kosnoff J, Yu K, Liu C, He B. Transcranial Focused Ultrasound to V5 Enhances Human Visual Motion Brain-Computer Interface by Modulating Feature-Based Attention. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.09.04.556252. [PMID: 37732253 PMCID: PMC10508752 DOI: 10.1101/2023.09.04.556252] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/22/2023]
Abstract
Paralysis affects roughly 1 in 50 Americans. While there is no cure for the condition, brain-computer interfaces (BCI) can allow users to control a device with their mind, bypassing the paralyzed region. Non-invasive BCIs still have high error rates, which is hypothesized to be reduced with concurrent targeted neuromodulation. This study examines whether transcranial focused ultrasound (tFUS) modulation can improve BCI outcomes, and what the underlying mechanism of action might be through high-density electroencephalography (EEG)-based source imaging (ESI) analyses. V5-targeted tFUS significantly reduced the error for the BCI speller task. ESI analyses showed significantly increased theta activity in the tFUS condition at both V5 and downstream the dorsal visual processing pathway. Correlation analysis indicates that the dorsal processing pathway connection was preserved during tFUS stimulation, whereas extraneous connections were severed. These results suggest that V5-targeted tFUS' mechanism of action is to raise the brain's feature-based attention to visual motion.
Collapse
|
8
|
Wang P, Zhou Y, Li Z, Huang S, Zhang D. Neural Decoding of Chinese Sign Language With Machine Learning for Brain-Computer Interfaces. IEEE Trans Neural Syst Rehabil Eng 2021; 29:2721-2732. [PMID: 34932480 DOI: 10.1109/tnsre.2021.3137340] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Limb motion decoding is an important part of brain-computer interface (BCI) research. Among the limb motion, sign language not only contains rich semantic information and abundant maneuverable actions but also provides different executable commands. However, many researchers focus on decoding the gross motor skills, such as the decoding of ordinary motor imagery or simple upper limb movements. Here we explored the neural features and decoding of Chinese sign language from electroencephalograph (EEG) signal with motor imagery and motor execution. Sign language not only contains rich semantic information, but also has abundant maneuverable actions, and provides us with more different executable commands. In this paper, twenty subjects were instructed to perform movement execution and movement imagery based on Chinese sign language. Seven classifiers are employed to classify the selected features of sign language EEG. L1 regularization is used to learn and select features that contain more information from the mean, power spectral density, sample entropy, and brain network connectivity. The best average classification accuracy of the classifier is 89.90% (imagery sign language is 83.40%). These results have shown the feasibility of decoding between different sign languages. The source location reveals that the neural circuits involved in sign language are related to the visual contact area and the pre-movement area. Experimental evaluation shows that the proposed decoding strategy based on sign language can obtain outstanding classification results, which provides a certain reference value for the subsequent research of limb decoding based on sign language.
Collapse
|
9
|
Randolph AB, Petter SC, Storey VC, Jackson MM. Context‐aware
user profiles to improve media synchronicity for individuals with severe motor disabilities. INFORMATION SYSTEMS JOURNAL 2021. [DOI: 10.1111/isj.12337] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
Affiliation(s)
- Adriane B. Randolph
- Information Systems and Security Kennesaw State University Kennesaw Georgia USA
| | | | - Veda C. Storey
- Computer Information Systems Georgia State University Atlanta Georgia USA
| | - Melody M. Jackson
- College of Computing Georgia Institute of Technology Atlanta Georgia USA
| |
Collapse
|
10
|
Anderson JR, Betts S, Fincham JM, Hope R, Walsh MW. Reconstructing fine-grained cognition from brain activity. Neuroimage 2020; 221:116999. [DOI: 10.1016/j.neuroimage.2020.116999] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2020] [Revised: 05/04/2020] [Accepted: 05/26/2020] [Indexed: 11/26/2022] Open
|
11
|
An X, Zhou X, Zhong W, Liu S, Li X, Ming D. Weighted Subject-Semi-Independent ERP-based Brain-Computer Interface. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:2969-2972. [PMID: 33018629 DOI: 10.1109/embc44109.2020.9176683] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Subject-independent brain-computer interfaces (SI-BCIs) which require no calibration process, are increasingly affect researchers in BCI field. The efficiencies (accuracies), however, were not satisfying till now. In this paper, we proposed a weighted subject-semi-independent classification method (WSSICM) for ERP based BCI system in which a few blocks data of target subject were used. 47 participants were attended in this study. We compared the accuracies of proposed method with traditional subject-specific classification method(SSCM) which used 15 blocks data of target subject. The averaged accuracies were 95.2% for the WSSICM at 5 blocks and 95.7% for the SSCM at 15 blocks. The accuracies of two method did not show significant difference (p-value=0.652). The method we proposed in this paper which could reduce the calibration time can be used for future BCI systems.
Collapse
|
12
|
Huggins JE, Guger C, Aarnoutse E, Allison B, Anderson CW, Bedrick S, Besio W, Chavarriaga R, Collinger JL, Do AH, Herff C, Hohmann M, Kinsella M, Lee K, Lotte F, Müller-Putz G, Nijholt A, Pels E, Peters B, Putze F, Rupp R, Schalk G, Scott S, Tangermann M, Tubig P, Zander T. Workshops of the Seventh International Brain-Computer Interface Meeting: Not Getting Lost in Translation. BRAIN-COMPUTER INTERFACES 2019; 6:71-101. [PMID: 33033729 PMCID: PMC7539697 DOI: 10.1080/2326263x.2019.1697163] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2019] [Accepted: 10/30/2019] [Indexed: 12/11/2022]
Abstract
The Seventh International Brain-Computer Interface (BCI) Meeting was held May 21-25th, 2018 at the Asilomar Conference Grounds, Pacific Grove, California, United States. The interactive nature of this conference was embodied by 25 workshops covering topics in BCI (also called brain-machine interface) research. Workshops covered foundational topics such as hardware development and signal analysis algorithms, new and imaginative topics such as BCI for virtual reality and multi-brain BCIs, and translational topics such as clinical applications and ethical assumptions of BCI development. BCI research is expanding in the diversity of applications and populations for whom those applications are being developed. BCI applications are moving toward clinical readiness as researchers struggle with the practical considerations to make sure that BCI translational efforts will be successful. This paper summarizes each workshop, providing an overview of the topic of discussion, references for additional information, and identifying future issues for research and development that resulted from the interactions and discussion at the workshop.
Collapse
Affiliation(s)
- Jane E Huggins
- Department of Physical Medicine and Rehabilitation, Department of Biomedical Engineering, Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan, United States, 325 East Eisenhower, Room 3017; Ann Arbor, Michigan 48108-5744
| | - Christoph Guger
- g.tec medical engineering GmbH/Guger Technologies OG, Austria, Sierningstrasse 14, 4521 Schiedlberg, Austria
| | - Erik Aarnoutse
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| | - Brendan Allison
- Dept. of Cognitive Science, Mail Code 0515, University of California at San Diego, La Jolla, United States
| | - Charles W Anderson
- Department of Computer Science, Molecular, Cellular and Integrative Neurosience Program, Colorado State University, Fort Collins, CO 80523
| | - Steven Bedrick
- Center for Spoken Language Understanding, Oregon Health & Science University, Portland, OR 97239
| | - Walter Besio
- Department of Electrical, Computer, & Biomedical Engineering and Interdisciplinary Neuroscience Program, University of Rhode Island, Kingston, Rhode Island, USA, CREmedical Corp. Kingston, Rhode Island, USA
| | - Ricardo Chavarriaga
- Defitech Chair in Brain-Machine Interface (CNBI), Center for Neuroprosthetics, Ecole Polytechnique Fédérale de Lausanne - EPFL, Switzerland
| | - Jennifer L Collinger
- University of Pittsburgh, Department of Physical Medicine and Rehabilitation, VA Pittsburgh Healthcare System, Department of Veterans Affairs, 3520 5th Ave, Pittsburgh, PA, 15213
| | - An H Do
- UC Irvine Brain Computer Interface Lab, Department of Neurology, University of California, Irvine
| | - Christian Herff
- School of Mental Health and Neuroscience, Maastricht University, Maastricht, The Netherlands
| | - Matthias Hohmann
- Max Planck Institute for Intelligent Systems, Department for Empirical Inference, Max-Planck-Ring 4, 72074 Tübingen, Germany
| | - Michelle Kinsella
- Oregon Health & Science University, Institute on Development & Disability, 707 SW Gaines St, #1290, Portland, OR 97239
| | - Kyuhwa Lee
- Swiss Federal Institute of Technology in Lausanne-EPFL
| | - Fabien Lotte
- Inria Bordeaux Sud-Ouest, LaBRI (Univ. Bordeaux/CNRS/Bordeaux INP), 200 avenue de la vieille tour, 33405, Talence Cedex, France
| | | | - Anton Nijholt
- Faculty EEMCS, University of Twente, Enschede, The Netherlands
| | - Elmar Pels
- UMC Utrecht Brain Center, Department of Neurology & Neurosurgery, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands
| | - Betts Peters
- Oregon Health & Science University, Institute on Development & Disability, 707 SW Gaines St, #1290, Portland, OR 97239
| | - Felix Putze
- University of Bremen, Germany, Cognitive Systems Lab, University of Bremen, Enrique-Schmidt-Straße 5 (Cartesium), 28359 Bremen
| | - Rüdiger Rupp
- Spinal Cord Injury Center, Heidelberg University Hospital
| | - Gerwin Schalk
- National Center for Adaptive Neurotechnologies, Wadsworth Center, NYS Dept. of Health, Dept. of Neurology, Albany Medical College, Dept. of Biomed. Sci., State Univ. of New York at Albany, Center for Medical Sciences 2003, 150 New Scotland Avenue, Albany, New York 12208
| | - Stephanie Scott
- Department of Media Communications, Colorado State University, Fort Collins, CO 80523
| | - Michael Tangermann
- Brain State Decoding Lab, Cluster of Excellence BrainLinks-BrainTools, Computer Science Dept., University of Freiburg, Germany, Autonomous Intelligent Systems Lab, Computer Science Dept., University of Freiburg, Germany
| | - Paul Tubig
- Department of Philosophy, Center for Neurotechnology, University of Washington, Savery Hall, Room 361, Seattle, WA 98195
| | - Thorsten Zander
- Team PhyPA, Biological Psychology and Neuroergonomics, Technische Universität Berlin, Berlin, Germany, 7 Zander Laboratories B.V., Amsterdam, The Netherlands
| |
Collapse
|
13
|
Renton AI, Mattingley JB, Painter DR. Optimising non-invasive brain-computer interface systems for free communication between naïve human participants. Sci Rep 2019; 9:18705. [PMID: 31822715 PMCID: PMC6904487 DOI: 10.1038/s41598-019-55166-y] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2019] [Accepted: 11/22/2019] [Indexed: 12/22/2022] Open
Abstract
Free communication is one of the cornerstones of modern civilisation. While manual keyboards currently allow us to interface with computers and manifest our thoughts, a next frontier is communication without manual input. Brain-computer interface (BCI) spellers often achieve this by decoding patterns of neural activity as users attend to flickering keyboard displays. To date, the highest performing spellers report typing rates of ~10.00 words/minute. While impressive, these rates are typically calculated for experienced users repetitively typing single phrases. It is therefore not clear whether naïve users are able to achieve such high rates with the added cognitive load of genuine free communication, which involves continuously generating and spelling novel words and phrases. In two experiments, we developed an open-source, high-performance, non-invasive BCI speller and examined its feasibility for free communication. The BCI speller required users to focus their visual attention on a flickering keyboard display, thereby producing unique cortical activity patterns for each key, which were decoded using filter-bank canonical correlation analysis. In Experiment 1, we tested whether seventeen naïve users could maintain rapid typing during prompted free word association. We found that information transfer rates were indeed slower during this free communication task than during typing of a cued character sequence. In Experiment 2, we further evaluated the speller's efficacy for free communication by developing a messaging interface, allowing users to engage in free conversation. The results showed that free communication was possible, but that information transfer was reduced by voluntary textual corrections and turn-taking during conversation. We evaluated a number of factors affecting the suitability of BCI spellers for free communication, and make specific recommendations for improving classification accuracy and usability. Overall, we found that developing a BCI speller for free communication requires a focus on usability over reduced character selection time, and as such, future performance appraisals should be based on genuine free communication scenarios.
Collapse
Affiliation(s)
- Angela I Renton
- Queensland Brain Institute, The University of Queensland, St Lucia, 4072, Australia.
| | - Jason B Mattingley
- Queensland Brain Institute, The University of Queensland, St Lucia, 4072, Australia
- School of Psychology, The University of Queensland, St Lucia, 4072, Australia
- Canadian Institute for Advanced Research (CIFAR), Toronto, Canada
| | - David R Painter
- School of Psychology, The University of Queensland, St Lucia, 4072, Australia
| |
Collapse
|
14
|
Fernández-Rodríguez Á, Velasco-Álvarez F, Medina-Juliá MT, Ron-Angevin R. Evaluation of emotional and neutral pictures as flashing stimuli using a P300 brain-computer interface speller. J Neural Eng 2019; 16:056024. [PMID: 31382248 DOI: 10.1088/1741-2552/ab386d] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Previous works have reported that complex emotional and visual stimuli can increase the amplitude of the P300 brain potential. Thus, the aim of the present work is to assess these kinds of images in a P300 brain-computer interface (BCI) speller as flashing stimuli. APPROACH Twenty-three volunteers controlled four spellers with different sets of flashing stimuli: flashing letters, neutral pictures (NP), emotional pleasant pictures (EPP) and emotional unpleasant pictures (EUP). MAIN RESULTS The sets of pictures showed a higher performance than the letters in accuracy and information transfer rate. These results were supported by the analysis of the P300 signal, where the picture sets offered the greatest amplitudes. The NP and EPP sets were the best evaluated in the subjective questionnaire. SIGNIFICANCE In short, despite the fact that the effect of emotional stimuli could not be observed in the performance metrics, picture sets have offered a high performance and should be considered in future proposals for visual P300-based BCI applications.
Collapse
|
15
|
Gembler F, Stawicki P, Saboor A, Volosyak I. Dynamic time window mechanism for time synchronous VEP-based BCIs-Performance evaluation with a dictionary-supported BCI speller employing SSVEP and c-VEP. PLoS One 2019; 14:e0218177. [PMID: 31194817 PMCID: PMC6564540 DOI: 10.1371/journal.pone.0218177] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2018] [Accepted: 05/28/2019] [Indexed: 11/18/2022] Open
Abstract
Brain-Computer Interfaces (BCIs) based on visual evoked potentials (VEPs) allow high communication speeds and accuracies. The fastest speeds can be achieved if targets are identified in a synchronous way (i.e., after a pre-set time period the system will produce a command output). The duration a target needs to be fixated on until the system classifies an output command affects the overall system performance. Hence, extracting a data window dedicated for the classification is of critical importance for VEP-based BCIs. Secondly, unintentional fixation on a target could easily lead to its selection. For the practical usability of BCI applications it is desirable to distinguish between intentional and unintentional fixations. This can be achieved by using threshold-based target identification methods. The study explores personalized dynamic classification time windows for threshold-based time synchronous VEP BCIs. The proposed techniques were tested employing the SSVEP and the c-VEP paradigm. Spelling performance was evaluated using an 8-target dictionary-supported BCI utilizing an n-gram word prediction model. The performance of twelve healthy participants was assessed with the information transfer rate (ITR) and accuracy. All participants completed sentence spelling tasks, reaching average accuracies of 94% and 96.3% for the c-VEP and the SSVEP paradigm, respectively. Average ITRs around 57 bpm were achieved for both paradigms.
Collapse
Affiliation(s)
- Felix Gembler
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany
| | - Piotr Stawicki
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany
| | - Abdul Saboor
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany
| | - Ivan Volosyak
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany
- * E-mail:
| |
Collapse
|
16
|
A Novel Dictionary-Driven Mental Spelling Application Based on Code-Modulated Visual Evoked Potentials. COMPUTERS 2019. [DOI: 10.3390/computers8020033] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Brain–computer interfaces (BCIs) based on code-modulated visual evoked potentials (c-VEPs) typically utilize a synchronous approach to identify targets (i.e., after preset time periods the system produces command outputs). Hence, users have only a limited amount of time to fixate a desired target. This hinders the usage of more complex interfaces, as these require the BCI to distinguish between intentional and unintentional fixations. In this article, we investigate a dynamic sliding window mechanism as well as the implementation of software-based stimulus synchronization to enable the threshold-based target identification for the c-VEP paradigm. To further improve the usability of the system, an ensemble-based classification strategy was investigated. In addition, a software-based approach for stimulus on-set determination is proposed, which allows for an easier setup of the system, as it reduces additional hardware dependencies. The methods were tested with an eight-target spelling application utilizing an n-gram word prediction model. The performance of eighteen participants without disabilities was tested; all participants completed word- and sentence spelling tasks using the c-VEP BCI with a mean information transfer rate (ITR) of 75.7 and 57.8 bpm, respectively.
Collapse
|
17
|
Vo K, Pham T, Nguyen DN, Kha HH, Dutkiewicz E. Subject-Independent ERP-Based Brain-Computer Interfaces. IEEE Trans Neural Syst Rehabil Eng 2019; 26:719-728. [PMID: 29641376 DOI: 10.1109/tnsre.2018.2810332] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Brain-computer interfaces (BCIs) are desirable for people to express their thoughts, especially those with profound disabilities in communication. The classification of brain patterns for each different subject requires an extensively time-consuming learning stage specific to that person, in order to reach satisfactory accuracy performance. The training session could also be infeasible for disabled patients as they may not fully understand the training instructions. In this paper, we propose a unified classification scheme based on ensemble classifier, dynamic stopping, and adaptive learning. We apply this scheme on the P300-based BCI, with the subject-independent manner, where no learning session is required for new experimental users. According to our theoretical analysis and empirical results, the harmonized integration of these three methods can significantly boost up the average accuracy from 75.00% to 91.26%, while at the same time reduce the average spelling time from 12.62 to 6.78 iterations, approximately to two-fold faster. The experiments were conducted on a large public dataset which had been used in other related studies. Direct comparisons between our work with the others' are also reported in details.
Collapse
|
18
|
Pedrozo Campos Antunes T, Souza Bulle de Oliveira A, Hudec R, Brusque Crocetta T, Ferreira de Lima Antão JY, de Almeida Barbosa RT, Guarnieri R, Massetti T, Garner DM, de Abreu LC. Assistive technology for communication of older adults: a systematic review. Aging Ment Health 2019; 23:417-427. [PMID: 29451399 DOI: 10.1080/13607863.2018.1426718] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/23/2023]
Abstract
OBJECTIVE Describe the use of assistive technology to enhance communication opportunities for older adults. METHODS A systematic review was conducted in two databases, PubMed and Web of Science, by using two different searches in each. The search was limited to original articles, in English language, including people aged 60 years and older that used any type of assistive technology for communication. The articles found in the initial search were filtered by title, abstracts and the remaining articles were fully read. RESULTS Eighteen studies were included in this review after the reading of full-texts. Most of the studies included apparently healthy participants with communication limitations due to aging related changes and the others included people with some pathology that prevent them from normal communication. CONCLUSION Four categories of assistive technology were identified: assistive technology for people with speech problems; robot or videoconferencing systems; Information and Communication Technologies and, other types of assistive technology for communication, such as hearing aids and scrapbooks. Assistive technology for communication of older adults is not only used by people with disabilities that prevent them from usual communication. They are mostly for older adults without a pathological communication problem.
Collapse
Affiliation(s)
- Thaiany Pedrozo Campos Antunes
- a Laboratório de Delineamento de Estudos e Escrita Científica da Faculdade de Medicina do ABC. Santo André , Sao Paulo , Brazil.,b Faculty of Electrical Engineering, Department of Multimedia and Information-Communication Technology, University of Zilina, Zilina , Slovakia
| | | | - Robert Hudec
- b Faculty of Electrical Engineering, Department of Multimedia and Information-Communication Technology, University of Zilina, Zilina , Slovakia
| | - Tania Brusque Crocetta
- a Laboratório de Delineamento de Estudos e Escrita Científica da Faculdade de Medicina do ABC. Santo André , Sao Paulo , Brazil.,d Faculty of Management Science and Informatics, Department of Mathematical Methods and Operations Research, University of Zilina, Zilina , Slovakia
| | | | - Renata Thais de Almeida Barbosa
- a Laboratório de Delineamento de Estudos e Escrita Científica da Faculdade de Medicina do ABC. Santo André , Sao Paulo , Brazil
| | - Regiani Guarnieri
- a Laboratório de Delineamento de Estudos e Escrita Científica da Faculdade de Medicina do ABC. Santo André , Sao Paulo , Brazil
| | - Thais Massetti
- e Departamento De Pós-graduação em Ciências da Reabilitação , Universidade de São Paulo , Brazil
| | - David M Garner
- f Faculty of Health and Life Sciences, Cardiorespiratory Research Group, Department of Biological and Medical Sciences, Oxford Brookes University , Oxford , United Kingdom
| | - Luiz Carlos de Abreu
- a Laboratório de Delineamento de Estudos e Escrita Científica da Faculdade de Medicina do ABC. Santo André , Sao Paulo , Brazil.,g Faculty of Public Health , University of Sao Paulo , Sao Paulo , Brazil.,h Department of Environmental Health, Harvard T.H. Chan School of Public Health , Boston , USA
| |
Collapse
|
19
|
Peters B, Higger M, Quivira F, Bedrick S, Dudy S, Eddy B, Kinsella M, Memmott T, Wiedrick J, Fried-Oken M, Erdogmus D, Oken B. Effects of simulated visual acuity and ocular motility impairments on SSVEP brain-computer interface performance: An experiment with Shuffle Speller. BRAIN-COMPUTER INTERFACES 2018; 5:58-72. [PMID: 30895198 DOI: 10.1080/2326263x.2018.1504662] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
Individuals with severe speech and physical impairments may have concomitant visual acuity impairments (VAI) or ocular motility impairments (OMI) impacting visual BCI use. We report on the use of the Shuffle Speller typing interface for an SSVEP BCI copy-spelling task under three conditions: simulated VAI, simulated OMI, and unimpaired vision. To mitigate the effect of visual impairments, we introduce a method that adaptively selects a user-specific trial length to maximize expected information transfer rate (ITR); expected ITR is shown to closely approximate the rate of correct letter selections. All participants could type under the unimpaired and simulated VAI conditions, with no significant differences in typing accuracy or speed. Most participants (31 of 37) could not type under the simulated OMI condition; some achieved high accuracy but with slower typing speeds. Reported workload and discomfort were low, and satisfaction high, under the unimpaired and simulated VAI conditions. Implications and future directions to examine effect of visual impairment on BCI use is discussed.
Collapse
Affiliation(s)
- Betts Peters
- Institute on Development & Disability, Oregon Health & Science University, Portland, OR
| | - Matt Higger
- Psychiatry Neuroimaging Laboratory, Brigham and Women's Hospital and Harvard Medical School, Boston, MA.,Electrical & Computer Engineering, Northeastern University, Boston, MA
| | - Fernando Quivira
- Electrical & Computer Engineering, Northeastern University, Boston, MA
| | - Steven Bedrick
- Center for Spoken Language Understanding, Oregon Health & Science University, Portland, OR
| | - Shiran Dudy
- Center for Spoken Language Understanding, Oregon Health & Science University, Portland, OR
| | - Brandon Eddy
- Institute on Development & Disability, Oregon Health & Science University, Portland, OR
| | - Michelle Kinsella
- Institute on Development & Disability, Oregon Health & Science University, Portland, OR
| | - Tab Memmott
- Departments of Neurology, Behavioral Neuroscience, and Biomedical Engineering, Oregon Health & Science University, Portland, OR
| | - Jack Wiedrick
- Biostatistics & Design Program, Oregon Health & Science University, Portland, OR
| | - Melanie Fried-Oken
- Institute on Development & Disability, Oregon Health & Science University, Portland, OR
| | - Deniz Erdogmus
- Electrical & Computer Engineering, Northeastern University, Boston, MA
| | - Barry Oken
- Departments of Neurology, Behavioral Neuroscience, and Biomedical Engineering, Oregon Health & Science University, Portland, OR
| |
Collapse
|
20
|
Mainsah BO, Reeves G, Collins LM, Throckmorton CS. Optimizing the stimulus presentation paradigm design for the P300-based brain-computer interface using performance prediction. J Neural Eng 2018; 14:046025. [PMID: 28548052 DOI: 10.1088/1741-2552/aa7525] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
OBJECTIVE The role of a brain-computer interface (BCI) is to discern a user's intended message or action by extracting and decoding relevant information from brain signals. Stimulus-driven BCIs, such as the P300 speller, rely on detecting event-related potentials (ERPs) in response to a user attending to relevant or target stimulus events. However, this process is error-prone because the ERPs are embedded in noisy electroencephalography (EEG) data, representing a fundamental problem in communication of the uncertainty in the information that is received during noisy transmission. A BCI can be modeled as a noisy communication system and an information-theoretic approach can be exploited to design a stimulus presentation paradigm to maximize the information content that is presented to the user. However, previous methods that focused on designing error-correcting codes failed to provide significant performance improvements due to underestimating the effects of psycho-physiological factors on the P300 ERP elicitation process and a limited ability to predict online performance with their proposed methods. Maximizing the information rate favors the selection of stimulus presentation patterns with increased target presentation frequency, which exacerbates refractory effects and negatively impacts performance within the context of an oddball paradigm. An information-theoretic approach that seeks to understand the fundamental trade-off between information rate and reliability is desirable. APPROACH We developed a performance-based paradigm (PBP) by tuning specific parameters of the stimulus presentation paradigm to maximize performance while minimizing refractory effects. We used a probabilistic-based performance prediction method as an evaluation criterion to select a final configuration of the PBP. MAIN RESULTS With our PBP, we demonstrate statistically significant improvements in online performance, both in accuracy and spelling rate, compared to the conventional row-column paradigm. SIGNIFICANCE By accounting for refractory effects, an information-theoretic approach can be exploited to significantly improve BCI performance across a wide range of performance levels.
Collapse
Affiliation(s)
- B O Mainsah
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, United States of America
| | | | | | | |
Collapse
|
21
|
A Multi-Context Character Prediction Model for a Brain-Computer Interface. PROCEEDINGS OF THE CONFERENCE. ASSOCIATION FOR COMPUTATIONAL LINGUISTICS. NORTH AMERICAN CHAPTER. MEETING 2018; 2018:72-77. [PMID: 33937917 DOI: 10.18653/v1/w18-1210] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
Brain-computer interfaces and other augmentative and alternative communication devices introduce language-modeing challenges distinct from other character-entry methods. In particular, the acquired signal of the EEG (electroencephalogram) signal is noisier, which, in turn, makes the user intent harder to decipher. In order to adapt to this condition, we propose to maintain ambiguous history for every time step, and to employ, apart from the character language model, word information to produce a more robust prediction system. We present preliminary results that compare this proposed Online-Context Language Model (OCLM) to current algorithms that are used in this type of setting. Evaluations on both perplexity and predictive accuracy demonstrate promising results when dealing with ambiguous histories in order to provide to the front end a distribution of the next character the user might type.
Collapse
|
22
|
Rezeika A, Benda M, Stawicki P, Gembler F, Saboor A, Volosyak I. Brain-Computer Interface Spellers: A Review. Brain Sci 2018; 8:brainsci8040057. [PMID: 29601538 PMCID: PMC5924393 DOI: 10.3390/brainsci8040057] [Citation(s) in RCA: 145] [Impact Index Per Article: 24.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2018] [Revised: 03/16/2018] [Accepted: 03/27/2018] [Indexed: 12/14/2022] Open
Abstract
A Brain-Computer Interface (BCI) provides a novel non-muscular communication method via brain signals. A BCI-speller can be considered as one of the first published BCI applications and has opened the gate for many advances in the field. Although many BCI-spellers have been developed during the last few decades, to our knowledge, no reviews have described the different spellers proposed and studied in this vital field. The presented speller systems are categorized according to major BCI paradigms: P300, steady-state visual evoked potential (SSVEP), and motor imagery (MI). Different BCI paradigms require specific electroencephalogram (EEG) signal features and lead to the development of appropriate Graphical User Interfaces (GUIs). The purpose of this review is to consolidate the most successful BCI-spellers published since 2010, while mentioning some other older systems which were built explicitly for spelling purposes. We aim to assist researchers and concerned individuals in the field by illustrating the highlights of different spellers and presenting them in one review. It is almost impossible to carry out an objective comparison between different spellers, as each has its variables, parameters, and conditions. However, the gathered information and the provided taxonomy about different BCI-spellers can be helpful, as it could identify suitable systems for first-hand users, as well as opportunities of development and learning from previous studies for BCI researchers.
Collapse
Affiliation(s)
- Aya Rezeika
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Mihaly Benda
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Piotr Stawicki
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Felix Gembler
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Abdul Saboor
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| | - Ivan Volosyak
- Faculty of Technology and Bionics, Rhine-Waal University of Applied Sciences, 47533 Kleve, Germany.
| |
Collapse
|
23
|
Pfeiffer T, Knight RT, Rose G. Hidden Markov model based continuous decoding of finger movements with prior knowledge incorporation using bi-gram models. Biomed Phys Eng Express 2018. [DOI: 10.1088/2057-1976/aa99f3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
24
|
Herrera-Arcos G, Tamez-Duque J, Acosta-De-Anda EY, Kwan-Loo K, de-Alba M, Tamez-Duque U, Contreras-Vidal JL, Soto R. Modulation of Neural Activity during Guided Viewing of Visual Art. Front Hum Neurosci 2017; 11:581. [PMID: 29249949 PMCID: PMC5714858 DOI: 10.3389/fnhum.2017.00581] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2017] [Accepted: 11/17/2017] [Indexed: 11/13/2022] Open
Abstract
Mobile Brain-Body Imaging (MoBI) technology was deployed to record multi-modal data from 209 participants to examine the brain's response to artistic stimuli at the Museo de Arte Contemporáneo (MARCO) in Monterrey, México. EEG signals were recorded as the subjects walked through the exhibit in guided groups of 6-8 people. Moreover, guided groups were either provided with an explanation of each art piece (Guided-E), or given no explanation (Guided-NE). The study was performed using portable Muse (InteraXon, Inc, Toronto, ON, Canada) headbands with four dry electrodes located at AF7, AF8, TP9, and TP10. Each participant performed a baseline (BL) control condition devoid of artistic stimuli and selected his/her favorite piece of art (FP) during the guided tour. In this study, we report data related to participants' demographic information and aesthetic preference as well as effects of art viewing on neural activity (EEG) in a select subgroup of 18-30 year-old subjects (Nc = 25) that generated high-quality EEG signals, on both BL and FP conditions. Dependencies on gender, sensor placement, and presence or absence of art explanation were also analyzed. After denoising, clustering of spectral EEG models was used to identify neural patterns associated with BL and FP conditions. Results indicate statistically significant suppression of beta band frequencies (15-25 Hz) in the prefrontal electrodes (AF7 and AF8) during appreciation of subjects' favorite painting, compared to the BL condition, which was significantly different from EEG responses to non-favorite paintings (NFP). No significant differences in brain activity in relation to the presence or absence of explanation during exhibit tours were found. Moreover, a frontal to posterior asymmetry in neural activity was observed, for both BL and FP conditions. These findings provide new information about frequency-related effects of preferred art viewing in brain activity, and support the view that art appreciation is independent of the artists' intent or original interpretation and related to the individual message that viewers themselves provide to each piece.
Collapse
Affiliation(s)
- Guillermo Herrera-Arcos
- Tecnológico de Monterrey, National Robotics Laboratory, School of Engineering and Sciences, Monterrey, Mexico
| | | | - Elsa Y Acosta-De-Anda
- Tecnológico de Monterrey, National Robotics Laboratory, School of Engineering and Sciences, Monterrey, Mexico
| | - Kevin Kwan-Loo
- Tecnológico de Monterrey, National Robotics Laboratory, School of Engineering and Sciences, Monterrey, Mexico
| | - Mayra de-Alba
- Tecnológico de Monterrey, National Robotics Laboratory, School of Engineering and Sciences, Monterrey, Mexico.,INDI Ingeniería y Diseño S.A.P.I. de C.V., Monterrey, Mexico
| | | | - Jose L Contreras-Vidal
- Tecnológico de Monterrey, National Robotics Laboratory, School of Engineering and Sciences, Monterrey, Mexico.,Laboratory for Non-invasive Brain-Machine Interface Systems, Department of Electrical and Computer Engineering, University of Houston, Houston, TX, United States
| | - Rogelio Soto
- Tecnológico de Monterrey, National Robotics Laboratory, School of Engineering and Sciences, Monterrey, Mexico
| |
Collapse
|
25
|
Pu X, Guo H, Chen J, Wang X, Xi Y, Hu C, Wang ZL. Eye motion triggered self-powered mechnosensational communication system using triboelectric nanogenerator. SCIENCE ADVANCES 2017; 3:e1700694. [PMID: 28782029 PMCID: PMC5533541 DOI: 10.1126/sciadv.1700694] [Citation(s) in RCA: 164] [Impact Index Per Article: 23.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/09/2017] [Accepted: 06/16/2017] [Indexed: 05/18/2023]
Abstract
Mechnosensational human-machine interfaces (HMIs) can greatly extend communication channels between human and external devices in a natural way. The mechnosensational HMIs based on biopotential signals have been developing slowly owing to the low signal-to-noise ratio and poor stability. In eye motions, the corneal-retinal potential caused by hyperpolarization and depolarization is very weak. However, the mechanical micromotion of the skin around the corners of eyes has never been considered as a good trigger signal source. We report a novel triboelectric nanogenerator (TENG)-based micromotion sensor enabled by the coupling of triboelectricity and electrostatic induction. By using an indium tin oxide electrode and two opposite tribomaterials, the proposed flexible and transparent sensor is capable of effectively capturing eye blink motion with a super-high signal level (~750 mV) compared with the traditional electrooculogram approach (~1 mV). The sensor is fixed on a pair of glasses and applied in two real-time mechnosensational HMIs-the smart home control system and the wireless hands-free typing system with advantages of super-high sensitivity, stability, easy operation, and low cost. This TENG-based micromotion sensor is distinct and unique in its fundamental mechanism, which provides a novel design concept for intelligent sensor technique and shows great potential application in mechnosensational HMIs.
Collapse
Affiliation(s)
- Xianjie Pu
- Department of Applied Physics, Chongqing University, Chongqing 400044, P. R. China
| | - Hengyu Guo
- Department of Applied Physics, Chongqing University, Chongqing 400044, P. R. China
- Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, National Center for Nanoscience and Technology, Beijing 100083, P. R. China
| | - Jie Chen
- Department of Applied Physics, Chongqing University, Chongqing 400044, P. R. China
| | - Xue Wang
- Department of Applied Physics, Chongqing University, Chongqing 400044, P. R. China
| | - Yi Xi
- Department of Applied Physics, Chongqing University, Chongqing 400044, P. R. China
| | - Chenguo Hu
- Department of Applied Physics, Chongqing University, Chongqing 400044, P. R. China
- Corresponding author. (C.H.); (Z.L.W.)
| | - Zhong Lin Wang
- Beijing Institute of Nanoenergy and Nanosystems, Chinese Academy of Sciences, National Center for Nanoscience and Technology, Beijing 100083, P. R. China
- Corresponding author. (C.H.); (Z.L.W.)
| |
Collapse
|
26
|
Iljina O, Derix J, Schirrmeister RT, Schulze-Bonhage A, Auer P, Aertsen A, Ball T. Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication. BRAIN-COMPUTER INTERFACES 2017. [DOI: 10.1080/2326263x.2017.1330611] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
- Olga Iljina
- GRK 1624 ‘Frequency effects in language’, University of Freiburg, Freiburg, Germany
- Department of German Linguistics, University of Freiburg, Freiburg, Germany
- Hermann Paul School of Linguistics, University of Freiburg, Germany
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
- Neurobiology and Biophysics, Faculty of Biology, University of Freiburg, Freiburg, Germany
| | - Johanna Derix
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
- Translational Neurotechnology Lab, Department of Neurosurgery, University Medical Center Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Robin Tibor Schirrmeister
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
- Translational Neurotechnology Lab, Department of Neurosurgery, University Medical Center Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| | - Andreas Schulze-Bonhage
- Epilepsy Center, Department of Neurosurgery, University Medical Center Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
| | - Peter Auer
- GRK 1624 ‘Frequency effects in language’, University of Freiburg, Freiburg, Germany
- Department of German Linguistics, University of Freiburg, Freiburg, Germany
- Hermann Paul School of Linguistics, University of Freiburg, Germany
- Freiburg Institute for Advanced Studies (FRIAS), University of Freiburg, Freiburg, Germany
| | - Ad Aertsen
- Neurobiology and Biophysics, Faculty of Biology, University of Freiburg, Freiburg, Germany
- Bernstein Center Freiburg, University of Freiburg, Germany
| | - Tonio Ball
- BrainLinks-BrainTools, University of Freiburg, Freiburg, Germany
- Translational Neurotechnology Lab, Department of Neurosurgery, University Medical Center Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany
| |
Collapse
|
27
|
Speier W, Deshpande A, Cui L, Chandravadia N, Roberts D, Pouratian N. A comparison of stimulus types in online classification of the P300 speller using language models. PLoS One 2017; 12:e0175382. [PMID: 28406932 PMCID: PMC5391014 DOI: 10.1371/journal.pone.0175382] [Citation(s) in RCA: 23] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2016] [Accepted: 03/06/2017] [Indexed: 11/18/2022] Open
Abstract
The P300 Speller is a common brain-computer interface communication system. There are many parallel lines of research underway to overcome the system's low signal to noise ratio and thereby improve performance, including using famous face stimuli and integrating language information into the classifier. While both have been shown separately to provide significant improvements, the two methods have not yet been implemented together to demonstrate that the improvements are complimentary. The goal of this study is therefore twofold. First, we aim to compare the famous faces stimulus paradigm with an existing alternative stimulus paradigm currently used in commercial systems (i.e., character inversion). Second, we test these methods with language model integration to assess whether different optimization approaches can be combined to further improve BCI communication. In offline analysis using a previously published particle filter method, famous faces stimuli yielded superior results to both standard and inverting stimuli. In online trials using the particle filter method, all 10 subjects achieved a higher selection rate when using the famous faces flashing paradigm than when using inverting flashes. The improvements achieved by these methods are therefore complementary and a combination yields superior results to either method implemented individually when tested in healthy subjects.
Collapse
Affiliation(s)
- William Speier
- Department of Neurosurgery, University of California, Los Angeles, Los Angeles, CA, United States of America
| | - Aniket Deshpande
- Department of Bioengineering, University of California, Los Angeles, Los Angeles, CA, United States of America
| | - Lucy Cui
- Neuroscience Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA, United States of America
| | - Nand Chandravadia
- Neuroscience Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA, United States of America
| | - Dustin Roberts
- Department of Neurosurgery, University of California, Los Angeles, Los Angeles, CA, United States of America
| | - Nader Pouratian
- Department of Neurosurgery, University of California, Los Angeles, Los Angeles, CA, United States of America
- Department of Bioengineering, University of California, Los Angeles, Los Angeles, CA, United States of America
- Neuroscience Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA, United States of America
- Brain Research Institute, University of California, Los Angeles, Los Angeles, CA, United States of America
| |
Collapse
|
28
|
Speier W, Chandravadia N, Roberts D, Pendekanti S, Pouratian N. Online BCI Typing using Language Model Classifiers by ALS Patients in their Homes. BRAIN-COMPUTER INTERFACES 2016; 4:114-121. [PMID: 29051907 DOI: 10.1080/2326263x.2016.1252143] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
Abstract
The P300 speller is a common brain-computer interface system that can provide a means of communication for patients with amyotrophic lateral sclerosis (ALS). Recent studies have shown that incorporating language information in signal classification can improve system performance, but they have largely been tested on healthy volunteers in a laboratory setting. The goal of this study was to demonstrate the functionality of the P300 speller system with language models when used by ALS patients in their homes. Six ALS patients with functional ratings ranging from two to 28 participated in this study. All subjects had improved offline performance when using a language model and five subjects were able to type at least six characters per minute with over 84% accuracy in online sessions. The results of this study indicate that the improvements in performance using language models in the P300 speller translate into the ALS population, which could help to make it a viable assistive device.
Collapse
Affiliation(s)
- William Speier
- Department of Neurosurgery, University of California, Los Angeles, USA
| | - Nand Chandravadia
- Neuroscience Interdepartmental Program, University of California, Los Angeles, USA
| | - Dustin Roberts
- Department of Neurosurgery, University of California, Los Angeles, USA
| | - S Pendekanti
- Neuroscience Interdepartmental Program, University of California, Los Angeles, USA
| | - Nader Pouratian
- Department of Neurosurgery, University of California, Los Angeles, USA.,Neuroscience Interdepartmental Program, University of California, Los Angeles, USA.,Department of Bioengineering, University of California, Los Angeles, USA.,Brain Research Institute, University of California, Los Angeles, USA
| |
Collapse
|