The WIN-speller: a new intuitive auditory brain-computer interface spelling application

Neural decoding of semantic concepts: A systematic literature review

Objective Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding. Approach We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity. Results Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area. Significance Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.

Poststroke Cognitive Impairment Research Progress on Application of Brain-Computer Interface

Brain-computer interfaces (BCIs), a new type of rehabilitation technology, pick up nerve cell signals, identify and classify their activities, and convert them into computer-recognized instructions. This technique has been widely used in the rehabilitation of stroke patients in recent years and appears to promote motor function recovery after stroke. At present, the application of BCI in poststroke cognitive impairment is increasing, which is a common complication that also affects the rehabilitation process. This paper reviews the promise and potential drawbacks of using BCI to treat poststroke cognitive impairment, providing a solid theoretical basis for the application of BCI in this area.

[Possibilities of the brain-computer interface in the correction of post-stroke cognitive impairments]

In recent years, brain-computer interfaces have been widely used in neurorehabilitation, and an extensive database of results from clinical studies conducted around the world has been accumulated, demonstrating their effectiveness in restoring motor function after a stroke. Currently, their use in post-stroke cognitive impairment is expanding. This article discusses the potential and prospects for using brain-computer interfaces for the treatment of cognitive disorders, reviews the experience of using it, presents the results of clinical studies in stroke patients, evaluates the possibilities of using this technology, describes the prospects, new directions of work on studying its effects.

Decoding of semantic categories of imagined concepts of animals and tools in fNIRS

Objective.Semantic decoding refers to the identification of semantic concepts from recordings of an individual's brain activity. It has been previously reported in functional magnetic resonance imaging and electroencephalography. We investigate whether semantic decoding is possible with functional near-infrared spectroscopy (fNIRS). Specifically, we attempt to differentiate between the semantic categories of animals and tools. We also identify suitable mental tasks for potential brain-computer interface (BCI) applications.Approach.We explore the feasibility of a silent naming task, for the first time in fNIRS, and propose three novel intuitive mental tasks based on imagining concepts using three sensory modalities: visual, auditory, and tactile. Participants are asked to visualize an object in their minds, imagine the sounds made by the object, and imagine the feeling of touching the object. A general linear model is used to extract hemodynamic responses that are then classified via logistic regression in a univariate and multivariate manner.Main results.We successfully classify all tasks with mean accuracies of 76.2% for the silent naming task, 80.9% for the visual imagery task, 72.8% for the auditory imagery task, and 70.4% for the tactile imagery task. Furthermore, we show that consistent neural representations of semantic categories exist by applying classifiers across tasks.Significance.These findings show that semantic decoding is possible in fNIRS. The study is the first step toward the use of semantic decoding for intuitive BCI applications for communication.

Brain-Computer Interfaces in Neurorecovery and Neurorehabilitation

Recent advances in brain-computer interface technology to restore and rehabilitate neurologic function aim to enable persons with disabling neurologic conditions to communicate, interact with the environment, and achieve other key activities of daily living and personal goals. Here we evaluate the principles, benefits, challenges, and future directions of brain-computer interfaces in the context of neurorehabilitation. We then explore the clinical translation of these technologies and propose an approach to facilitate implementation of brain-computer interfaces for persons with neurologic disease.

Stimulus modality influences session-to-session transfer of training effects in auditory and tactile streaming-based P300 brain-computer interfaces

Despite recent successes, patients suffering from locked-in syndrome (LIS) still struggle to communicate using vision-independent brain–computer interfaces (BCIs). In this study, we compared auditory and tactile BCIs, regarding training effects and cross-stimulus-modality transfer effects, when switching between stimulus modalities. We utilized a streaming-based P300 BCI, which was developed as a low workload approach to prevent potential BCI-inefficiency. We randomly assigned 20 healthy participants to two groups. The participants received three sessions of training either using an auditory BCI or using a tactile BCI. In an additional fourth session, BCI versions were switched to explore possible cross-stimulus-modality transfer effects. Both BCI versions could be operated successfully in the first session by the majority of the participants, with the tactile BCI being experienced as more intuitive. Significant training effects were found mostly in the auditory BCI group and strong evidence for a cross-stimulus-modality transfer occurred for the auditory training group that switched to the tactile version but not vice versa. All participants were able to control at least one BCI version, suggesting that the investigated paradigms are generally feasible and merit further research into their applicability with LIS end-users. Individual preferences regarding stimulus modality should be considered.

Classification of Drowsiness Levels Based on a Deep Spatio-Temporal Convolutional Bidirectional LSTM Network Using Electroencephalography Signals

Non-invasive brain-computer interfaces (BCI) have been developed for recognizing human mental states with high accuracy and for decoding various types of mental conditions. In particular, accurately decoding a pilot's mental state is a critical issue as more than 70% of aviation accidents are caused by human factors, such as fatigue or drowsiness. In this study, we report the classification of not only two mental states (i.e., alert and drowsy states) but also five drowsiness levels from electroencephalogram (EEG) signals. To the best of our knowledge, this approach is the first to classify drowsiness levels in detail using only EEG signals. We acquired EEG data from ten pilots in a simulated night flight environment. For accurate detection, we proposed a deep spatio-temporal convolutional bidirectional long short-term memory network (DSTCLN) model. We evaluated the classification performance using Karolinska sleepiness scale (KSS) values for two mental states and five drowsiness levels. The grand-averaged classification accuracies were 0.87 (±0.01) and 0.69 (±0.02), respectively. Hence, we demonstrated the feasibility of classifying five drowsiness levels with high accuracy using deep learning.

Using brain-computer interfaces: a scoping review of studies employing social research methods

Background

The rapid expansion of research on Brain-Computer Interfaces (BCIs) is not only due to the promising solutions offered for persons with physical impairments. There is also a heightened need for understanding BCIs due to the challenges regarding ethics presented by new technology, especially in its impact on the relationship between man and machine. Here we endeavor to present a scoping review of current studies in the field to gain insight into the complexity of BCI use. By examining studies related to BCIs that employ social research methods, we seek to demonstrate the multitude of approaches and concerns from various angles in considering the social and human impact of BCI technology.

Methods

For this scoping review of research on BCIs’ social and ethical implications, we systematically analyzed six databases, encompassing the fields of medicine, psychology, and the social sciences, in order to identify empirical studies on BCIs. The search yielded 73 publications that employ quantitative, qualitative, or mixed methods.

Results

Of the 73 publications, 71 studies address the user perspective. Some studies extend to consideration of other BCI stakeholders such as medical technology experts, caregivers, or health care professionals. The majority of the studies employ quantitative methods. Recurring themes across the studies examined were general user opinion towards BCI, central technical or social issues reported, requests/demands made by users of the technology, the potential/future of BCIs, and ethical aspects of BCIs.

Conclusions

Our findings indicate that while technical aspects of BCIs such as usability or feasibility are being studied extensively, comparatively little in-depth research has been done on the self-image and self-experience of the BCI user. In general there is also a lack of focus or examination of the caregiver’s perspective.

Electronic supplementary material

The online version of this article (10.1186/s12910-019-0354-1) contains supplementary material, which is available to authorized users.

Toward Automated Cochlear Implant Fitting Procedures Based on Event-Related Potentials

OBJECTIVES

Cochlear implants (CIs) restore hearing to the profoundly deaf by direct electrical stimulation of the auditory nerve. To provide an optimal electrical stimulation pattern the CI must be individually fitted to each CI user. To date, CI fitting is primarily based on subjective feedback from the user. However, not all CI users are able to provide such feedback, for example, small children. This study explores the possibility of using the electroencephalogram (EEG) to objectively determine if CI users are able to hear differences in tones presented to them, which has potential applications in CI fitting or closed loop systems.

DESIGN

Deviant and standard stimuli were presented to 12 CI users in an active auditory oddball paradigm. The EEG was recorded in two sessions and classification of the EEG data was performed with shrinkage linear discriminant analysis. Also, the impact of CI artifact removal on classification performance and the possibility to reuse a trained classifier in future sessions were evaluated.

RESULTS

Overall, classification performance was above chance level for all participants although performance varied considerably between participants. Also, artifacts were successfully removed from the EEG without impairing classification performance. Finally, reuse of the classifier causes only a small loss in classification performance.

CONCLUSIONS

Our data provide first evidence that EEG can be automatically classified on single-trial basis in CI users. Despite the slightly poorer classification performance over sessions, classifier and CI artifact correction appear stable over successive sessions. Thus, classifier and artifact correction weights can be reused without repeating the set-up procedure in every session, which makes the technique easier applicable. With our present data, we can show successful classification of event-related cortical potential patterns in CI users. In the future, this has the potential to objectify and automate parts of CI fitting procedures.

A Multifunctional Brain-Computer Interface Intended for Home Use: An Evaluation with Healthy Participants and Potential End Users with Dry and Gel-Based Electrodes

Current brain-computer interface (BCIs) software is often tailored to the needs of scientists and technicians and therefore complex to allow for versatile use. To facilitate home use of BCIs a multifunctional P300 BCI with a graphical user interface intended for non-expert set-up and control was designed and implemented. The system includes applications for spelling, web access, entertainment, artistic expression and environmental control. In addition to new software, it also includes new hardware for the recording of electroencephalogram (EEG) signals. The EEG system consists of a small and wireless amplifier attached to a cap that can be equipped with gel-based or dry contact electrodes. The system was systematically evaluated with a healthy sample, and targeted end users of BCI technology, i.e., people with a varying degree of motor impairment tested the BCI in a series of individual case studies. Usability was assessed in terms of effectiveness, efficiency and satisfaction. Feedback of users was gathered with structured questionnaires. Two groups of healthy participants completed an experimental protocol with the gel-based and the dry contact electrodes (N = 10 each). The results demonstrated that all healthy participants gained control over the system and achieved satisfactory to high accuracies with both gel-based and dry electrodes (average error rates of 6 and 13%). Average satisfaction ratings were high, but certain aspects of the system such as the wearing comfort of the dry electrodes and design of the cap, and speed (in both groups) were criticized by some participants. Six potential end users tested the system during supervised sessions. The achieved accuracies varied greatly from no control to high control with accuracies comparable to that of healthy volunteers. Satisfaction ratings of the two end-users that gained control of the system were lower as compared to healthy participants. The advantages and disadvantages of the BCI and its applications are discussed and suggestions are presented for improvements to pave the way for user friendly BCIs intended to be used as assistive technology by persons with severe paralysis.

Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation

Brain-machine interfaces (BMIs) combine methods, approaches, and concepts derived from neurophysiology, computer science, and engineering in an effort to establish real-time bidirectional links between living brains and artificial actuators. Although theoretical propositions and some proof of concept experiments on directly linking the brains with machines date back to the early 1960s, BMI research only took off in earnest at the end of the 1990s, when this approach became intimately linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals of BMIs are 1) to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations to severely disabled patients. Over the past decade, a wide range of BMI applications have emerged, which considerably expanded these original goals. BMI studies have shown neural control over the movements of robotic and virtual actuators that enact both upper and lower limb functions. Furthermore, BMIs have also incorporated ways to deliver sensory feedback, generated from external actuators, back to the brain. BMI research has been at the forefront of many neurophysiological discoveries, including the demonstration that, through continuous use, artificial tools can be assimilated by the primate brain's body schema. Work on BMIs has also led to the introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term continuous BMI use has been recently implicated with the induction of partial neurological recovery in spinal cord injury patients.

A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives

A new Brain-Computer Interface (BCI) technique, which is called a hybrid BCI, has recently been proposed to address the limitations of conventional single BCI system. Although some hybrid BCI studies have shown promising results, the field of hybrid BCI is still in its infancy and there is much to be done. Especially, since the hybrid BCI systems are so complicated and complex, it is difficult to understand the constituent and role of a hybrid BCI system at a glance. Also, the complicated and complex systems make it difficult to evaluate the usability of the systems. We systematically reviewed and analyzed the current state-of-the-art hybrid BCI studies, and proposed a systematic taxonomy for classifying the types of hybrid BCIs with multiple taxonomic criteria. After reviewing 74 journal articles, hybrid BCIs could be categorized with respect to 1) the source of brain signals, 2) the characteristics of the brain signal, and 3) the characteristics of operation in each system. In addition, we exhaustively reviewed recent literature on usability of BCIs. To identify the key evaluation dimensions of usability, we focused on task and measurement characteristics of BCI usability. We classified and summarized 31 BCI usability journal articles according to task characteristics (type and description of task) and measurement characteristics (subjective and objective measures). Afterwards, we proposed usability dimensions for BCI and hybrid BCI systems according to three core-constructs: Satisfaction, effectiveness, and efficiency with recommendations for further research. This paper can help BCI researchers, even those who are new to the field, can easily understand the complex structure of the hybrid systems at a glance. Recommendations for future research can also be helpful in establishing research directions and gaining insight in how to solve ergonomics and HCI design issues surrounding BCI and hybrid BCI systems by usability evaluation.

An Evaluation of Training with an Auditory P300 Brain-Computer Interface for the Japanese Hiragana Syllabary

Gaze-independent brain-computer interfaces (BCIs) are a possible communication channel for persons with paralysis. We investigated if it is possible to use auditory stimuli to create a BCI for the Japanese Hiragana syllabary, which has 46 Hiragana characters. Additionally, we investigated if training has an effect on accuracy despite the high amount of different stimuli involved. Able-bodied participants (N = 6) were asked to select 25 syllables (out of fifty possible choices) using a two step procedure: First the consonant (ten choices) and then the vowel (five choices). This was repeated on 3 separate days. Additionally, a person with spinal cord injury (SCI) participated in the experiment. Four out of six healthy participants reached Hiragana syllable accuracies above 70% and the information transfer rate increased from 1.7 bits/min in the first session to 3.2 bits/min in the third session. The accuracy of the participant with SCI increased from 12% (0.2 bits/min) to 56% (2 bits/min) in session three. Reliable selections from a 10 × 5 matrix using auditory stimuli were possible and performance is increased by training. We were able to show that auditory P300 BCIs can be used for communication with up to fifty symbols. This enables the use of the technology of auditory P300 BCIs with a variety of applications.

Interfacing brain with computer to improve communication and rehabilitation after brain damage

Communication and control of the external environment can be provided via brain-computer interfaces (BCIs) to replace a lost function in persons with severe diseases and little or no chance of recovery of motor abilities (ie, amyotrophic lateral sclerosis, brainstem stroke). BCIs allow to intentionally modulate brain activity, to train specific brain functions, and to control prosthetic devices, and thus, this technology can also improve the outcome of rehabilitation programs in persons who have suffered from a central nervous system injury (ie, stroke leading to motor or cognitive impairment). Overall, the BCI researcher is challenged to interact with people with severe disabilities and professionals in the field of neurorehabilitation. This implies a deep understanding of the disabled condition on the one hand, and it requires extensive knowledge on the physiology and function of the human brain on the other. For these reasons, a multidisciplinary approach and the continuous involvement of BCI users in the design, development, and testing of new systems are desirable. In this chapter, we will focus on noninvasive EEG-based systems and their clinical applications, highlighting crucial issues to foster BCI translation outside laboratories to eventually become a technology usable in real-life realm.