Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

Recent progress on smart lower prosthetic limbs: a comprehensive review on using EEG and fNIRS devices in rehabilitation

The global rise in lower limb amputation cases necessitates advancements in prosthetic limb technology to enhance the quality of life for affected patients. This review paper explores recent advancements in the integration of EEG and fNIRS modalities for smart lower prosthetic limbs for rehabilitation applications. The paper synthesizes current research progress, focusing on the synergy between brain-computer interfaces and neuroimaging technologies to enhance the functionality and user experience of lower limb prosthetics. The review discusses the potential of EEG and fNIRS in decoding neural signals, enabling more intuitive and responsive control of prosthetic devices. Additionally, the paper highlights the challenges, innovations, and prospects associated with the incorporation of these neurotechnologies in the field of rehabilitation. The insights provided in this review contribute to a deeper understanding of the evolving landscape of smart lower prosthetic limbs and pave the way for more effective and user-friendly solutions in the realm of neurorehabilitation.

Evaluating the Feasibility of Visual Imagery for an EEG-Based Brain-Computer Interface

Visual imagery, or the mental simulation of visual information from memory, could serve as an effective control paradigm for a brain-computer interface (BCI) due to its ability to directly convey the user's intention with many natural ways of envisioning an intended action. However, multiple initial investigations into using visual imagery as a BCI control strategies have been unable to fully evaluate the capabilities of true spontaneous visual mental imagery. One major limitation in these prior works is that the target image is typically displayed immediately preceding the imagery period. This paradigm does not capture spontaneous mental imagery as would be necessary in an actual BCI application but something more akin to short-term retention in visual working memory. Results from the present study show that short-term visual imagery following the presentation of a specific target image provides a stronger, more easily classifiable neural signature in EEG than spontaneous visual imagery from long-term memory following an auditory cue for the image. We also show that short-term visual imagery and visual perception share commonalities in the most predictive electrodes and spectral features. However, visual imagery received greater influence from frontal electrodes whereas perception was mostly confined to occipital electrodes. This suggests that visual perception is primarily driven by sensory information whereas visual imagery has greater contributions from areas associated with memory and attention. This work provides the first direct comparison of short-term and long-term visual imagery tasks and provides greater insight into the feasibility of using visual imagery as a BCI control strategy.

Low frequency oscillations reflect neurovascular coupling and disappear after cerebral death

Spectrum power analysis in the low frequency oscillations (LFO) region of functional near infrared spectroscopy (fNIRS) is a promising method to deliver information about brain activation and therefore might be used for prognostication in patients with disorders of consciousness in the neurocritical care unit alongside with established methods. In this study, we measure the cortical hemodynamic response measured by fNIRS in the LFO region following auditory and somatosensory stimulation in healthy subjects. The significant hemodynamic reaction in the contralateral hemisphere correlation with the physiologic electric response suggests neurovascular coupling. In addition, we investigate power spectrum changes in steady state measurements of cerebral death patients and healthy subjects in the LFO region, the frequency of the heartbeat and respiration. The spectral power within the LFO region was lower in the patients with cerebral death compared to the healthy subjects, whereas there were no differences in spectral power for physiological activities such as heartbeat and respiration rate. This finding indicates the cerebral origin of our low frequency measurements. Therefore, LFO measurements are a potential method to detect brain activation in patients with disorders of consciousness and cerebral death. However, further studies in patients are needed to investigate its potential clinical use.

A novel brain-controlled prosthetic hand method integrating AR-SSVEP augmentation, asynchronous control, and machine vision assistance

Background and objective

The brain-computer interface (BCI) system based on steady-state visual evoked potentials (SSVEP) is expected to help disabled patients achieve alternative prosthetic hand assistance. However, the existing study still has some shortcomings in interaction aspects such as stimulus paradigm and control logic. The purpose of this study is to innovate the visual stimulus paradigm and asynchronous decoding/control strategy by integrating augmented reality technology, and propose an asynchronous pattern recognition algorithm, thereby improving the interaction logic and practical application capabilities of the prosthetic hand with the BCI system.

Methods

An asynchronous visual stimulus paradigm based on an augmented reality (AR) interface was proposed in this paper, in which there were 8 control modes, including Grasp, Put down, Pinch, Point, Fist, Palm push, Hold pen, and Initial. According to the attentional orienting characteristics of the paradigm, a novel asynchronous pattern recognition algorithm that combines center extended canonical correlation analysis and support vector machine (Center-ECCA-SVM) was proposed. Then, this study proposed an intelligent BCI system switch based on a deep learning object detection algorithm (YOLOv4) to improve the level of user interaction. Finally, two experiments were designed to test the performance of the brain-controlled prosthetic hand system and its practical performance in real scenarios.

Results

Under the AR paradigm of this study, compared with the liquid crystal display (LCD) paradigm, the average SSVEP spectrum amplitude of multiple subjects increased by 17.41%, and the signal-noise ratio (SNR) increased by 3.52%. The average stimulus pattern recognition accuracy was 96.71 ± 3.91%, which was 2.62% higher than the LCD paradigm. Under the data analysis time of 2s, the Center-ECCA-SVM classifier obtained 94.66 ± 3.87% and 97.40 ± 2.78% asynchronous pattern recognition accuracy under the Normal metric and the Tolerant metric, respectively. And the YOLOv4-tiny model achieves a speed of 25.29fps and a 96.4% confidence in the prosthetic hand in real-time detection. Finally, the brain-controlled prosthetic hand helped the subjects to complete 4 kinds of daily life tasks in the real scene, and the time-consuming were all within an acceptable range, which verified the effectiveness and practicability of the system.

Conclusion

This research is based on improving the user interaction level of the prosthetic hand with the BCI system, and has made improvements in the SSVEP paradigm, asynchronous pattern recognition, interaction, and control logic. Furthermore, it also provides support for BCI areas for alternative prosthetic control, and movement disorder rehabilitation programs.

A scoping review of gaze and eye tracking-based control methods for assistive robotic arms

Background: Assistive Robotic Arms are designed to assist physically disabled people with daily activities. Existing joysticks and head controls are not applicable for severely disabled people such as people with Locked-in Syndrome. Therefore, eye tracking control is part of ongoing research. The related literature spans many disciplines, creating a heterogeneous field that makes it difficult to gain an overview. Objectives: This work focuses on ARAs that are controlled by gaze and eye movements. By answering the research questions, this paper provides details on the design of the systems, a comparison of input modalities, methods for measuring the performance of these controls, and an outlook on research areas that gained interest in recent years. Methods: This review was conducted as outlined in the PRISMA 2020 Statement. After identifying a wide range of approaches in use the authors decided to use the PRISMA-ScR extension for a scoping review to present the results. The identification process was carried out by screening three databases. After the screening process, a snowball search was conducted. Results: 39 articles and 6 reviews were included in this article. Characteristics related to the system and study design were extracted and presented divided into three groups based on the use of eye tracking. Conclusion: This paper aims to provide an overview for researchers new to the field by offering insight into eye tracking based robot controllers. We have identified open questions that need to be answered in order to provide people with severe motor function loss with systems that are highly useable and accessible.

Unleashing the potential of fNIRS with machine learning: classification of fine anatomical movements to empower future brain-computer interface

In this study, we explore the potential of using functional near-infrared spectroscopy (fNIRS) signals in conjunction with modern machine-learning techniques to classify specific anatomical movements to increase the number of control commands for a possible fNIRS-based brain-computer interface (BCI) applications. The study focuses on novel individual finger-tapping, a well-known task in fNIRS and fMRI studies, but limited to left/right or few fingers. Twenty-four right-handed participants performed the individual finger-tapping task. Data were recorded by using sixteen sources and detectors placed over the motor cortex according to the 10-10 international system. The event's average oxygenated Δ HbO and deoxygenated Δ HbR hemoglobin data were utilized as features to assess the performance of diverse machine learning (ML) models in a challenging multi-class classification setting. These methods include LDA, QDA, MNLR, XGBoost, and RF. A new DL-based model named "Hemo-Net" has been proposed which consists of multiple parallel convolution layers with different filters to extract the features. This paper aims to explore the efficacy of using fNRIS along with ML/DL methods in a multi-class classification task. Complex models like RF, XGBoost, and Hemo-Net produce relatively higher test set accuracy when compared to LDA, MNLR, and QDA. Hemo-Net has depicted a superior performance achieving the highest test set accuracy of 76%, however, in this work, we do not aim at improving the accuracies of models rather we are interested in exploring if fNIRS has the neural signatures to help modern ML/DL methods in multi-class classification which can lead to applications like brain-computer interfaces. Multi-class classification of fine anatomical movements, such as individual finger movements, is difficult to classify with fNIRS data. Traditional ML models like MNLR and LDA show inferior performance compared to the ensemble-based methods of RF and XGBoost. DL-based method Hemo-Net outperforms all methods evaluated in this study and demonstrates a promising future for fNIRS-based BCI applications.

Multimodal pre-screening can predict BCI performance variability: A novel subject-specific experimental scheme

BACKGROUND

Brain-computer interface (BCI) systems currently lack the required robustness for long-term daily use due to inter- and intra-subject performance variability. In this study, we propose a novel personalized scheme for a multimodal BCI system, primarily using functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), to identify, predict, and compensate for factors affecting competence-related and interfering factors associated with performance.

METHOD

11 (out of 13 recruited) participants, including five participants with motor deficits, completed four sessions on average. During the training sessions, the subjects performed a short pre-screening phase, followed by three variations of a novel visou-mental (VM) protocol. Features extracted from the pre-screening phase were used to construct predictive platforms using stepwise multivariate linear regression (MLR) models. In the test sessions, we employed a task-correction phase where our predictive models were used to predict the ideal task variation to maximize performance, followed by an interference-correction phase. We then investigated the associations between predicted and actual performances and evaluated the outcome of correction strategies.

RESULT

The predictive models resulted in respective adjusted R-squared values of 0.942, 0.724, and 0.939 for the first, second, and third variation of the task, respectively. The statistical analyses showed significant associations between the performances predicted by predictive models and the actual performances for the first two task variations, with rhos of 0.7289 (p-value = 0.011) and 0.6970 (p-value = 0.017), respectively. For 81.82 % of the subjects, the task/workload correction stage correctly determined which task variation provided the highest accuracy, with an average performance gain of 5.18 % when applying the correction strategies.

CONCLUSION

Our proposed method can lead to an integrated multimodal predictive framework to compensate for BCI performance variability, particularly, for people with severe motor deficits.

The clinical effects of brain-computer interface with robot on upper-limb function for post-stroke rehabilitation: a meta-analysis and systematic review

PURPOSE

Many recent clinical studies have suggested that the combination of brain-computer interfaces (BCIs) can induce neurological recovery and improvement in motor function. In this review, we performed a systematic review and meta-analysis to evaluate the clinical effects of BCI-robot systems.

METHODS

The articles published from January 2010 to December 2020 have been searched by using the databases (EMBASE, PubMed, CINAHL, EBSCO, Web of Science and manual search). The single-group studies were qualitatively described, and only the controlled-trial studies were included for the meta-analysis. The mean difference (MD) of Fugl-Meyer Assessment (FMA) scores were pooled and the random-effects model method was used to perform the meta-analysis. The PRISMA criteria were followed in current review.

RESULTS

A total of 897 records were identified, eight single-group studies and 11 controlled-trial studies were included in our review. The systematic analysis indicated that the BCI-robot systems had a significant improvement on motor function recovery. The meta-analysis showed there were no statistic differences between BCI-robot groups and robot groups, neither in the immediate effects nor long-term effects (p > 0.05).

CONCLUSION

The use of BCI-robot systems has significant improvement on the motor function recovery of hemiparetic upper-limb, and there is a sustaining effect. The meta-analysis showed no statistical difference between the experimental group (BCI-robot) and the control group (robot). However, there are a few shortcomings in the experimental design of existing studies, more clinical trials need to be conducted, and the experimental design needs to be more rigorous.Implications for RehabilitationIn this review, we evaluated the clinical effects of brain-computer interface with robot on upper-limb function for post-stroke rehabilitation. After we screened the databases, 19 articles were included in this review. These articles all clinical trial research, they all used non-invasive brain-computer interfaces and upper-limb robot.We conducted the systematic review with nine articles, the result indicated that the BCI-robot system had a significant improvement on motor function recovery. Eleven articles were included for the meta-analysis, the result showed there were no statistic differences between BCI-robot groups and robot groups, neither in the immediate effects nor long-term effects.We thought the result of meta-analysis which showed no statistic difference was probably caused by the heterogenicity of clinical trial designs of these articles.We thought the BCI-robot systems are promising strategies for post-stroke rehabilitation. And we gave several suggestions for further research: (1) The experimental design should be more rigorous, and describe the experimental designs in detail, especially the control group intervention, to make the experiment replicability. (2) New evaluation criteria need to be established, more objective assessment such as biomechanical assessment, fMRI should be utilised as the primary outcome. (3) More clinical studies with larger sample size, novel external devices, and BCI systems need to be conducted to investigate the differences between BCI-robot system and other interventions. (4) Further research could shift the focus to the patients who are in subacute stage, to explore if the early BCI training can make a positive impact on cerebral cortical recovery.

fNIRS-EEG BCIs for Motor Rehabilitation: A Review

Motor impairment has a profound impact on a significant number of individuals, leading to a substantial demand for rehabilitation services. Through brain-computer interfaces (BCIs), people with severe motor disabilities could have improved communication with others and control appropriately designed robotic prosthetics, so as to (at least partially) restore their motor abilities. BCI plays a pivotal role in promoting smoother communication and interactions between individuals with motor impairments and others. Moreover, they enable the direct control of assistive devices through brain signals. In particular, their most significant potential lies in the realm of motor rehabilitation, where BCIs can offer real-time feedback to assist users in their training and continuously monitor the brain's state throughout the entire rehabilitation process. Hybridization of different brain-sensing modalities, especially functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), has shown great potential in the creation of BCIs for rehabilitating the motor-impaired populations. EEG, as a well-established methodology, can be combined with fNIRS to compensate for the inherent disadvantages and achieve higher temporal and spatial resolution. This paper reviews the recent works in hybrid fNIRS-EEG BCIs for motor rehabilitation, emphasizing the methodologies that utilized motor imagery. An overview of the BCI system and its key components was introduced, followed by an introduction to various devices, strengths and weaknesses of different signal processing techniques, and applications in neuroscience and clinical contexts. The review concludes by discussing the possible challenges and opportunities for future development.

A global bibliometric and visualized analysis of gait analysis and artificial intelligence research from 1992 to 2022

Gait is an important basic function of human beings and an integral part of life. Many mental and physical abnormalities can cause noticeable differences in a person's gait. Abnormal gait can lead to serious consequences such as falls, limited mobility and reduced life satisfaction. Gait analysis, which includes joint kinematics, kinetics, and dynamic Electromyography (EMG) data, is now recognized as a clinically useful tool that can provide both quantifiable and qualitative information on performance to aid in treatment planning and evaluate its outcome. With the assistance of new artificial intelligence (AI) technology, the traditional medical environment has undergone great changes. AI has the potential to reshape medicine, making gait analysis more accurate, efficient and accessible. In this study, we analyzed basic information about gait analysis and AI articles that met inclusion criteria in the WoS Core Collection database from 1992-2022, and the VosViewer software was used for web visualization and keyword analysis. Through bibliometric and visual analysis, this article systematically introduces the research status of gait analysis and AI. We introduce the application of artificial intelligence in clinical gait analysis, which affects the identification and management of gait abnormalities found in various diseases. Machine learning (ML) and artificial neural networks (ANNs) are the most often utilized AI methods in gait analysis. By comparing the predictive capability of different AI algorithms in published studies, we evaluate their potential for gait analysis in different situations. Furthermore, the current challenges and future directions of gait analysis and AI research are discussed, which will also provide valuable reference information for investors in this field.

Spectrum-Enhanced TRCA (SE-TRCA): A novel approach for direction detection in SSVEP-based BCI

The Steady State Visual Evoked Potential (SSVEP) is a widely used component in BCIs due to its high noise resistance and low equipment requirements. Recently, a novel SSVEP-based paradigm has been introduced for direction detection, in which, unlike the common SSVEP paradigms that use several frequency stimuli, only one flickering stimulus is used and it makes direction detection very challenging. So far, only the CCA method has been used for direction detection using SSVEP component analysis. Since Canonical Correlation Analysis (CCA) has some limitations, a Task-Related Component Analysis (TRCA) based method has been introduced for feature extraction to improve the direction detection performance. Although these methods have been proven efficient, they do not utilize the latent frequency information in the EEG signal. Therefore, the performance of direction detection using SSVEP component analysis is still suboptimal. For further improvement, the TRCA-based algorithm is enhanced by incorporating frequency information and introducing Spectrum-Enhanced TRCA (SE-TRCA). SE-TRCA method can utilize frequency information in conjunction with spatial information by concatenating the EEG signal and its shifted version. Accordingly, the obtained spatio-spectral filters perform as a Finite Impulse Response (FIR) filter. To evaluate the proposed SE-TRCA method, two different sorts of datasets (1) a hybrid BCI dataset (including SSVEP component for direction detection) and (2) a pure benchmark SSVEP dataset (including SSVEP component for frequency detection) have been used. Our experiments showed that the accuracy of direction detection using the proposed SE-TRCA and TRCA approaches compared to CCA-based approach have been increased by 23.35% and 28.24%, respectively. Furthermore, the accuracy of character recognition obtained from integrating P300 and SSVEP components in CCA, TRCA, and SETRCA approaches are 54.01%, 56.02%, and 58.56%, on the hybrid dataset, respectively. The evaluation of the SE-TRCA method on the benchmark SSVEP dataset demonstrates that the SE-TRCA method outperforms both CCA and TRCA, particularly regarding frequency detection accuracy. In this specific dataset, the SE-TRCA method achieved an impressive frequency detection accuracy of 98.19% for a 3-s signal, surpassing the accuracies of TRCA and CCA, which were 97.91% and 90.47%, respectively. These results demonstrated that the TRCA-based approach is more efficient than the CCA approach to extracting spatial filters. Moreover, SE-TRCA, extracting both Spectral and spatial information from the EEG signal, can capture more discriminative features from the SSVEP component and increase the accuracy of classification. The results of this study emphasize the effectiveness of the proposed SE-TRCA approach across different SSVEP paradigms and tasks. These findings provide strong evidence for the method's ability to generalize well in SSVEP analysis.

The characteristics of brain network in patient with post-stroke depression under cognitive task condition

Objectives

Post-stroke depression (PSD) may be associated with the altered brain network property. This study aimed at exploring the brain network characteristics of PSD under the classic cognitive task, i.e., the oddball task, in order to promote our understanding of the pathogenesis and the diagnosis of PSD.

Methods

Nineteen stroke survivors with PSD and 18 stroke survivors with no PSD (non-PSD) were recruited. The functional near-infrared spectroscopy (fNIRS) covering the dorsolateral prefrontal cortex was recorded during the oddball task state and the resting state. The brain network characteristics were extracted using the graph theory and compared between the PSD and the non-PSD subjects. In addition, the classification performance between the PSD and non-PSD subjects was evaluated using features in the resting and the task state, respectively.

Results

Compared with the resting state, more brain network characteristics in the task state showed significant differences between the PSD and non-PSD groups, resulting in better classification performance. In the task state, the assortativity, clustering coefficient, characteristic path length, and local efficiency of the PSD subjects was larger compared with the non-PSD subjects while the global efficiency of the PSD subjects was smaller than that of the non-PSD subjects.

Conclusion

The altered brain network properties associated with PSD in the cognitive task state were more distinct compared with the resting state, and the ability of the brain network to resist attack and transmit information was reduced in PSD patients in the task state.

Significance

This study demonstrated the feasibility and superiority of investigating brain network properties in the task state for the exploration of the pathogenesis and new diagnosis methods for PSD.

Multi-Class Classification of Upper Limb Movements With Filter Bank Task-Related Component Analysis

The classification of limb movements can provide with control commands in non-invasive brain-computer interface. Previous studies on the classification of limb movements have focused on the classification of left/right limbs; however, the classification of different types of upper limb movements has often been ignored despite that it provides more active-evoked control commands in the brain-computer interface. Nevertheless, few machine learning method can be used as the state-of-the-art method in the multi-class classification of limb movements. This work focuses on the multi-class classification of upper limb movements and proposes the multi-class filter bank task-related component analysis (mFBTRCA) method, which consists of three steps: spatial filtering, similarity measuring and filter bank selection. The spatial filter, namely the task-related component analysis, is first used to remove noise from EEG signals. The canonical correlation measures the similarity of the spatial-filtered signals and is used for feature extraction. The correlation features are extracted from multiple low-frequency filter banks. The minimum-redundancy maximum-relevance selects the essential features from all the correlation features, and finally, the support vector machine is used to classify the selected features. The proposed method compared against previously used models is evaluated using two datasets. mFBTRCA achieved a classification accuracy of 0.4193 ± 0.0780 (7 classes) and 0.4032 ± 0.0714 (5 classes), respectively, which improves on the best accuracies achieved using the compared methods (0.3590 ± 0.0645 and 0.3159 ± 0.0736, respectively). The proposed method is expected to provide more control commands in the applications of non-invasive brain-computer interfaces.

Driving drowsiness detection using spectral signatures of EEG-based neurophysiology

Introduction: Drowsy driving is a significant factor causing dire road crashes and casualties around the world. Detecting it earlier and more effectively can significantly reduce the lethal aftereffects and increase road safety. As physiological conditions originate from the human brain, so neurophysiological signatures in drowsy and alert states may be investigated for this purpose. In this preface, A passive brain-computer interface (pBCI) scheme using multichannel electroencephalography (EEG) brain signals is developed for spatially localized and accurate detection of human drowsiness during driving tasks.

Methods: This pBCI modality acquired electrophysiological patterns of 12 healthy subjects from the prefrontal (PFC), frontal (FC), and occipital cortices (OC) of the brain. Neurological states are recorded using six EEG channels spread over the right and left hemispheres in the PFC, FC, and OC of the sleep-deprived subjects during simulated driving tasks. In post-hoc analysis, spectral signatures of the δ, θ, α, and β rhythms are extracted in terms of spectral band powers and their ratios with a temporal correlation over the complete span of the experiment. Minimum redundancy maximum relevance, Chi-square, and ReliefF feature selection methods are used and aggregated with a Z-score based approach for global feature ranking. The extracted drowsiness attributes are classified using decision trees, discriminant analysis, logistic regression, naïve Bayes, support vector machines, k-nearest neighbors, and ensemble classifiers. The binary classification results are reported with confusion matrix-based performance assessment metrics.

Results: In inter-classifier comparison, the optimized ensemble model achieved the best results of drowsiness classification with 85.6% accuracy and precision, 89.7% recall, 87.6% F1-score, 80% specificity, 70.3% Matthews correlation coefficient, 70.2% Cohen’s kappa score, and 91% area under the receiver operating characteristic curve with 76-ms execution time. In inter-channel comparison, the best results were obtained at the F8 electrode position in the right FC of the brain. The significance of all the results was validated with a p-value of less than 0.05 using statistical hypothesis testing methods.

Conclusions: The proposed scheme has achieved better results for driving drowsiness detection with the accomplishment of multiple objectives. The predictor importance approach has reduced the feature extraction cost and computational complexity is minimized with the use of conventional machine learning classifiers resulting in low-cost hardware and software requirements. The channel selection approach has spatially localized the most promising brain region for drowsiness detection with only a single EEG channel (F8) which reduces the physical intrusiveness in normal driving operation. This pBCI scheme has a good potential for practical applications requiring earlier, more accurate, and less disruptive drowsiness detection using the spectral information of EEG biosignals.

Functional Near-Infrared Spectroscopy as a Personalized Digital Healthcare Tool for Brain Monitoring

The sustained growth of digital healthcare in the field of neurology relies on portable and cost-effective brain monitoring tools that can accurately monitor brain function in real time. Functional near-infrared spectroscopy (fNIRS) is one such tool that has become popular among researchers and clinicians as a practical alternative to functional magnetic resonance imaging, and as a complementary tool to modalities such as electroencephalography. This review covers the contribution of fNIRS to the personalized goals of digital healthcare in neurology by identifying two major trends that drive current fNIRS research. The first major trend is multimodal monitoring using fNIRS, which allows clinicians to access more data that will help them to understand the interconnection between the cerebral hemodynamics and other physiological phenomena in patients. This allows clinicians to make an overall assessment of physical health to obtain a more-detailed and individualized diagnosis. The second major trend is that fNIRS research is being conducted with naturalistic experimental paradigms that involve multisensory stimulation in familiar settings. Cerebral monitoring of multisensory stimulation during dynamic activities or within virtual reality helps to understand the complex brain activities that occur in everyday life. Finally, the scope of future fNIRS studies is discussed to facilitate more-accurate assessments of brain activation and the wider clinical acceptance of fNIRS as a medical device for digital healthcare.

Exploring high-density corticomuscular networks after stroke to enable a hybrid Brain-Computer Interface for hand motor rehabilitation

BACKGROUND

Brain-Computer Interfaces (BCI) promote upper limb recovery in stroke patients reinforcing motor related brain activity (from electroencephalogaphy, EEG). Hybrid BCIs which include peripheral signals (electromyography, EMG) as control features could be employed to monitor post-stroke motor abnormalities. To ground the use of corticomuscular coherence (CMC) as a hybrid feature for a rehabilitative BCI, we analyzed high-density CMC networks (derived from multiple EEG and EMG channels) and their relation with upper limb motor deficit by comparing data from stroke patients with healthy participants during simple hand tasks.

METHODS

EEG (61 sensors) and EMG (8 muscles per arm) were simultaneously recorded from 12 stroke (EXP) and 12 healthy participants (CTRL) during simple hand movements performed with right/left (CTRL) and unaffected/affected hand (EXP, UH/AH). CMC networks were estimated for each movement and their properties were analyzed by means of indices derived ad-hoc from graph theory and compared among groups.

RESULTS

Between-group analysis showed that CMC weight of the whole brain network was significantly reduced in patients during AH movements. The network density was increased especially for those connections entailing bilateral non-target muscles. Such reduced muscle-specificity observed in patients was confirmed by muscle degree index (connections per muscle) which indicated a connections' distribution among non-target and contralateral muscles and revealed a higher involvement of proximal muscles in patients. CMC network properties correlated with upper-limb motor impairment as assessed by Fugl-Meyer Assessment and Manual Muscle Test in patients.

CONCLUSIONS

High-density CMC networks can capture motor abnormalities in stroke patients during simple hand movements. Correlations with upper limb motor impairment support their use in a BCI-based rehabilitative approach.

Identifying Thematics in a Brain-Computer Interface Research

This umbrella review is motivated to understand the shift in research themes on brain-computer interfacing (BCI) and it determined that a shift away from themes that focus on medical advancement and system development to applications that included education, marketing, gaming, safety, and security has occurred. The background of this review examined aspects of BCI categorisation, neuroimaging methods, brain control signal classification, applications, and ethics. The specific area of BCI software and hardware development was not examined. A search using One Search was undertaken and 92 BCI reviews were selected for inclusion. Publication demographics indicate the average number of authors on review papers considered was 4.2 ± 1.8. The results also indicate a rapid increase in the number of BCI reviews from 2003, with only three reviews before that period, two in 1972, and one in 1996. While BCI authors were predominantly Euro-American in early reviews, this shifted to a more global authorship, which China dominated by 2020-2022. The review revealed six disciplines associated with BCI systems: life sciences and biomedicine (n = 42), neurosciences and neurology (n = 35), and rehabilitation (n = 20); (2) the second domain centred on the theme of functionality: computer science (n = 20), engineering (n = 28) and technology (n = 38). There was a thematic shift from understanding brain function and modes of interfacing BCI systems to more applied research novel areas of research-identified surround artificial intelligence, including machine learning, pre-processing, and deep learning. As BCI systems become more invasive in the lives of "normal" individuals, it is expected that there will be a refocus and thematic shift towards increased research into ethical issues and the need for legal oversight in BCI application.

Investigating User Proficiency of Motor Imagery for EEG-Based BCI System to Control Simulated Wheelchair

The research on the electroencephalography (EEG)-based brain-computer interface (BCI) is widely utilized for wheelchair control. The ability of the user is one factor of BCI efficiency. Therefore, we focused on BCI tasks and protocols to yield high efficiency from the robust EEG features of individual users. This study proposes a task-based brain activity to gain the power of the alpha band, which included eyes closed for alpha response at the occipital area, attention to an upward arrow for alpha response at the frontal area, and an imagined left/right motor for alpha event-related desynchronization at the left/right motor cortex. An EPOC X neuroheadset was used to acquire the EEG signals. We also proposed user proficiency in motor imagery sessions with limb movement paradigms by recommending motor imagination tasks. Using the proposed system, we verified the feature extraction algorithms and command translation. Twelve volunteers participated in the experiment, and the conventional paradigm of motor imagery was used to compare the efficiencies. With utilized user proficiency in motor imagery, an average accuracy of 83.7% across the left and right commands was achieved. The recommended MI paradigm via user proficiency achieved an approximately 4% higher accuracy than the conventional MI paradigm. Moreover, the real-time control results of a simulated wheelchair revealed a high efficiency based on the time condition. The time results for the same task as the joystick-based control were still approximately three times longer. We suggest that user proficiency be used to recommend an individual MI paradigm for beginners. Furthermore, the proposed BCI system can be used for electric wheelchair control by people with severe disabilities.

fNIRS-based brain functional response to robot-assisted training for upper-limb in stroke patients with hemiplegia

Background

Robot-assisted therapy (RAT) has received considerable attention in stroke motor rehabilitation. Characteristics of brain functional response associated with RAT would provide a theoretical basis for choosing the appropriate protocol for a patient. However, the cortical response induced by RAT remains to be fully elucidated due to the lack of dynamic brain functional assessment tools.

Objective

To guide the implementation of clinical therapy, this study focused on the brain functional responses induced by RAT in patients with different degrees of motor impairment.

Methods

A total of 32 stroke patients were classified into a low score group (severe impairment, n = 16) and a high score group (moderate impairment, n = 16) according to the motor function of the upper limb and then underwent RAT training in assistive mode with simultaneous cerebral haemodynamic measurement by functional near-infrared spectroscopy (fNIRS). Functional connectivity (FC) and the hemisphere autonomy index (HAI) were calculated based on the wavelet phase coherence among fNIRS signals covering bilateral prefrontal, motor and occipital areas.

Results

Specific cortical network response related to RAT was observed in patients with unilateral moderate-to-severe motor deficits in the subacute stage. Compared with patients with moderate dysfunction, patients with severe impairment showed a wide range of significant FC responses in the bilateral hemispheres induced by RAT with the assistive mode, especially task-related involvement of ipsilesional supplementary motor areas.

Conclusion

Under assisted mode, RAT-related extensive cortical response in patients with severe dysfunction might contribute to brain functional organization during motor performance, which is considered the basic neural substrate of motor-related processes. In contrast, the limited cortical response related to RAT in patients with moderate dysfunction may indicate that the training intensity needs to be adjusted in time according to the brain functional state. fNIRS-based assessment of brain functional response assumes great importance for the customization of an appropriate protocol training in the clinical practice.

A Review of Brain Activity and EEG-Based Brain-Computer Interfaces for Rehabilitation Application

Patients with severe CNS injuries struggle primarily with their sensorimotor function and communication with the outside world. There is an urgent need for advanced neural rehabilitation and intelligent interaction technology to provide help for patients with nerve injuries. Recent studies have established the brain-computer interface (BCI) in order to provide patients with appropriate interaction methods or more intelligent rehabilitation training. This paper reviews the most recent research on brain-computer-interface-based non-invasive rehabilitation systems. Various endogenous and exogenous methods, advantages, limitations, and challenges are discussed and proposed. In addition, the paper discusses the communication between the various brain-computer interface modes used between severely paralyzed and locked patients and the surrounding environment, particularly the brain-computer interaction system utilizing exogenous (induced) EEG signals (such as P300 and SSVEP). This discussion reveals with an examination of the interface for collecting EEG signals, EEG components, and signal postprocessing. Furthermore, the paper describes the development of natural interaction strategies, with a focus on signal acquisition, data processing, pattern recognition algorithms, and control techniques.

EEG hybrid brain-computer interfaces: A scoping review applying an existing hybrid-BCI taxonomy and considerations for pediatric applications

Most hybrid brain-computer interfaces (hBCI) aim at improving the performance of single-input BCI. Many combinations are possible to configure an hBCI, such as using multiple brain input signals, different stimuli or more than one input system. Multiple studies have been done since 2010 where such interfaces have been tested and analyzed. Results and conclusions are promising but little has been discussed as to what is the best approach for the pediatric population, should they use hBCI as an assistive technology. Children might face greater challenges when using BCI and might benefit from less complex interfaces. Hence, in this scoping review we included 42 papers that developed hBCI systems for the purpose of control of assistive devices or communication software, and we analyzed them through the lenses of potential use in clinical settings and for children. We extracted taxonomic categories proposed in previous studies to describe the types of interfaces that have been developed. We also proposed interface characteristics that could be observed in different hBCI, such as type of target, number of targets and number of steps before selection. Then, we discussed how each of the extracted characteristics could influence the overall complexity of the system and what might be the best options for applications for children. Effectiveness and efficiency were also collected and included in the analysis. We concluded that the least complex hBCI interfaces might involve having a brain inputs and an external input, with a sequential role of operation, and visual stimuli. Those interfaces might also use a minimal number of targets of the strobic type, with one or two steps before the final selection. We hope this review can be used as a guideline for future hBCI developments and as an incentive to the design of interfaces that can also serve children who have motor impairments.

EEG-EMG coupling as a hybrid method for steering detection in car driving settings

Understanding mental processes in complex human behavior is a key issue in driving, representing a milestone for developing user-centered assistive driving devices. Here, we propose a hybrid method based on electroencephalographic (EEG) and electromyographic (EMG) signatures to distinguish left and right steering in driving scenarios. Twenty-four participants took part in the experiment consisting of recordings of 128-channel EEG and EMG activity from deltoids and forearm extensors in non-ecological and ecological steering tasks. Specifically, we identified the EEG mu rhythm modulation correlates with motor preparation of self-paced steering actions in the non-ecological task, while the concurrent EMG activity of the left (right) deltoids correlates with right (left) steering. Consequently, we exploited the mu rhythm de-synchronization resulting from the non-ecological task to detect the steering side using cross-correlation analysis with the ecological EMG signals. Results returned significant cross-correlation values showing the coupling between the non-ecological EEG feature and the muscular activity collected in ecological driving conditions. Moreover, such cross-correlation patterns discriminate the steering side earlier relative to the single EMG signal. This hybrid system overcomes the limitation of the EEG signals collected in ecological settings such as low reliability, accuracy, and adaptability, thus adding to the EMG the characteristic predictive power of the cerebral data. These results prove how it is possible to complement different physiological signals to control the level of assistance needed by the driver.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-021-09776-w.

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.

High-Frequency Vibrating Stimuli Using the Low-Cost Coin-Type Motors for SSSEP-Based BCI

Steady-state somatosensory-evoked potential- (SSSEP-) based brain-computer interfaces (BCIs) have been applied for assisting people with physical disabilities since it does not require gaze fixation or long-time training. Despite the advancement of various noninvasive electroencephalogram- (EEG-) based BCI paradigms, researches on SSSEP with the various frequency range and related classification algorithms are relatively unsettled. In this study, we investigated the feasibility of classifying the SSSEP within high-frequency vibration stimuli induced by a versatile coin-type eccentric rotating mass (ERM) motor. Seven healthy subjects performed selective attention (SA) tasks with vibration stimuli attached to the left and right index fingers. Three EEG feature extraction methods, followed by a support vector machine (SVM) classifier, have been tested: common spatial pattern (CSP), filter-bank CSP (FBCSP), and mutual information-based best individual feature (MIBIF) selection after the FBCSP. Consequently, the FBCSP showed the highest performance at $71.5\pm 2.5$ % for classifying the left and right-hand SA tasks than the other two methods (i.e., CSP and FBCSP-MIBIF). Based on our findings and approach, the high-frequency vibration stimuli using low-cost coin motors with the FBCSP-based feature selection can be potentially applied to developing practical SSSEP-based BCI systems.

Bimodal EEG-fNIRS in Neuroergonomics. Current Evidence and Prospects for Future Research

Neuroergonomics focuses on the brain signatures and associated mental states underlying behavior to design human-machine interfaces enhancing performance in the cognitive and physical domains. Brain imaging techniques such as functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) have been considered key methods for achieving this goal. Recent research stresses the value of combining EEG and fNIRS in improving these interface systems' mental state decoding abilities, but little is known about whether these improvements generalize over different paradigms and methodologies, nor about the potentialities for using these systems in the real world. We review 33 studies comparing mental state decoding accuracy between bimodal EEG-fNIRS and unimodal EEG and fNIRS in several subdomains of neuroergonomics. In light of these studies, we also consider the challenges of exploiting wearable versions of these systems in real-world contexts. Overall the studies reviewed suggest that bimodal EEG-fNIRS outperforms unimodal EEG or fNIRS despite major differences in their conceptual and methodological aspects. Much work however remains to be done to reach practical applications of bimodal EEG-fNIRS in naturalistic conditions. We consider these points to identify aspects of bimodal EEG-fNIRS research in which progress is expected or desired.

Temel Duygusal Durumların Hemodinamik Karşılıklarının Taşınabilir bir İşlevsel Yakın Kızılaltı Spektroskopi Sistemi ile Tanımlanması

Objective: The aim of this study was to test the feasibility of a functional near infrared spectroscopy (fNIRS) system for quantification of the similarities and differences in the spatial localization of cerebral hemodynamic activation induced by visual presentation of neutral, negative and positive valence emotional stimuli. Method: 13 healthy subjects viewed neutral, pleasant and unpleasant pictures from the International Affective Picture System (IAPS) database in a block design experiment while the prefrontal cortical hemodynamic changes induced by emotional stimuli were continuously recorded with a 20 channel fNIRS system that covered the forehead region. Results: Negative valence pictures induced higher hemodynamic activity in right lateralized regions involving dorsolateral and oribtofrontal cortex when compared to neutral and positive valence stimuli (pFDR

MCFHNet: Multi-Channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding

Motor Imagery-based Brain Computer Interface (MI-BCI) is a typical active BCI with a main focus on motor intention identification. Hybrid motor imagery (MI) decoding methods that based on multi-modal fusion of Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), especially deep learning-based methods, become popular in recent MI-BCI studies. However, the fusion strategy and network design in deep learning-based methods are complex. To solve this problem, we proposed the multi-channel fusion method (MCF) to simplify current fusion methods, and we designed a multi-channel fusion hybrid network (MCFHNet) based on MCF. MCFHNet combines depthwise convolutional layers, channel attention mechanism, and Bidirectional Long Short Term Memory (Bi-LSTM) layers, which enables strong capability of feature extraction in spatial and temporal domain. The comparison between MCFHNet and representative deep learning-based methods was performed on an open EEG-fNIRS dataset. We found the proposed method can yield superior performance (mean accuracy of 99.641 % in 5-fold cross validation of an intra-subject experiment). This work provides a new option for multi-modal MI decoding, which can be applied in the rehabilitation field based on hybrid BCI systems.

Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting

Virtual reality environments offer great opportunities to study the performance of brain-computer interfaces (BCIs) in real-world contexts. As real-world stimuli are typically multimodal, their neuronal integration elicits complex response patterns. To investigate the effect of additional auditory cues on the processing of visual information, we used virtual reality to mimic safety-related events in an industrial environment while we concomitantly recorded electroencephalography (EEG) signals. We simulated a box traveling on a conveyor belt system where two types of stimuli – an exploding and a burning box – interrupt regular operation. The recordings from 16 subjects were divided into two subsets, a visual-only and an audio-visual experiment. In the visual-only experiment, the response patterns for both stimuli elicited a similar pattern – a visual evoked potential (VEP) followed by an event-related potential (ERP) over the occipital-parietal lobe. Moreover, we found the perceived severity of the event to be reflected in the signal amplitude. Interestingly, the additional auditory cues had a twofold effect on the previous findings: The P1 component was significantly suppressed in the case of the exploding box stimulus, whereas the N2c showed an enhancement for the burning box stimulus. This result highlights the impact of multisensory integration on the performance of realistic BCI applications. Indeed, we observed alterations in the offline classification accuracy for a detection task based on a mixed feature extraction (variance, power spectral density, and discrete wavelet transform) and a support vector machine classifier. In the case of the explosion, the accuracy slightly decreased by –1.64% p. in an audio-visual experiment compared to the visual-only. Contrarily, the classification accuracy for the burning box increased by 5.58% p. when additional auditory cues were present. Hence, we conclude, that especially in challenging detection tasks, it is favorable to consider the potential of multisensory integration when BCIs are supposed to operate under (multimodal) real-world conditions.

A Unified Local–Global Feature Extraction Network for Human Gait Recognition Using Smartphone Sensors

Smartphone-based gait recognition has been considered a unique and promising technique for biometric-based identification. It is integrated with multiple sensors to collect inertial data while a person walks. However, captured data may be affected by several covariate factors due to variations of gait sequences such as holding loads, wearing types, shoe types, etc. Recent gait recognition approaches either work on global or local features, causing failure to handle these covariate-based features. To address these issues, a novel weighted multi-scale CNN (WMsCNN) architecture is designed to extract local to global features for boosting recognition accuracy. Specifically, a weight update sub-network (Ws) is proposed to increase or reduce the weights of features concerning their contribution to the final classification task. Thus, the sensitivity of these features toward the covariate factors decreases using the weight updated technique. Later, these features are fed to a fusion module used to produce global features for the overall classification. Extensive experiments have been conducted on four different benchmark datasets, and the demonstrated results of the proposed model are superior to other state-of-the-art deep learning approaches.

Unmanned Aerial Vehicle for Laser Based Biomedical Sensor Development and Examination of Device Trajectory

Controller design and signal processing for the control of air-vehicles have gained extreme importance while interacting with humans to form a brain–computer interface. This is because fewer commands need to be mapped into multiple controls. For our anticipated biomedical sensor for breath analysis, it is mandatory to provide medication to the patients on an urgent basis. To address this increasingly tense situation in terms of emergencies, we plan to design an unmanned vehicle that can aid spontaneously to monitor the person’s health, and help the physician spontaneously during the rescue mission. Simultaneously, that must be done in such a computationally efficient algorithm that the minimum amount of energy resources are consumed. For this purpose, we resort to an unmanned logistic air-vehicle which flies from the medical centre to the affected person. After obtaining restricted permission from the regional administration, numerous challenges are identified for this design. The device is able to lift a weight of 2 kg successfully which is required for most emergency medications, while choosing the smallest distance to the destination with the GPS. By recording the movement of the vehicle in numerous directions, the results deviate to a maximum of 2% from theoretical investigations. In this way, our biomedical sensor provides critical information to the physician, who is able to provide medication to the patient urgently. On account of reasonable supply of medicines to the destination in terms of weight and time, this experimentation has been rendered satisfactory by the relevant physicians in the vicinity.

Dual Passive Reactive Brain-Computer Interface: A Novel Approach to Human-Machine Symbiosis

The present study proposes a novel concept of neuroadaptive technology, namely a dual passive-reactive Brain-Computer Interface (BCI), that enables bi-directional interaction between humans and machines. We have implemented such a system in a realistic flight simulator using the NextMind classification algorithms and framework to decode pilots' intention (reactive BCI) and to infer their level of attention (passive BCI). Twelve pilots used the reactive BCI to perform checklists along with an anti-collision radar monitoring task that was supervised by the passive BCI. The latter simulated an automatic avoidance maneuver when it detected that pilots missed an incoming collision. The reactive BCI reached 100% classification accuracy with a mean reaction time of 1.6 s when exclusively performing the checklist task. Accuracy was up to 98.5% with a mean reaction time of 2.5 s when pilots also had to fly the aircraft and monitor the anti-collision radar. The passive BCI achieved a F1-score of 0.94. This first demonstration shows the potential of a dual BCI to improve human-machine teaming which could be applied to a variety of applications.

Involvement of the Rostromedial Prefrontal Cortex in Human-Robot Interaction: fNIRS Evidence From a Robot-Assisted Motor Task

Assistive exoskeleton robots are being widely applied in neurorehabilitation to improve upper-limb motor and somatosensory functions. During robot-assisted exercises, the central nervous system appears to highly attend to external information-processing (IP) to efficiently interact with robotic assistance. However, the neural mechanisms underlying this process remain unclear. The rostromedial prefrontal cortex (rmPFC) may be the core of the executive resource allocation that generates biases in the allocation of processing resources toward an external IP according to current behavioral demands. Here, we used functional near-infrared spectroscopy to investigate the cortical activation associated with executive resource allocation during a robot-assisted motor task. During data acquisition, participants performed a right-arm motor task using elbow flexion-extension movements in three different loading conditions: robotic assistive loading (ROB), resistive loading (RES), and non-loading (NON). Participants were asked to strive for kinematic consistency in their movements. A one-way repeated measures analysis of variance and general linear model-based methods were employed to examine task-related activity. We demonstrated that hemodynamic responses in the ventral and dorsal rmPFC were higher during ROB than during NON. Moreover, greater hemodynamic responses in the ventral rmPFC were observed during ROB than during RES. Increased activation in ventral and dorsal rmPFC subregions may be involved in the executive resource allocation that prioritizes external IP during human-robot interactions. In conclusion, these findings provide novel insights regarding the involvement of executive control during a robot-assisted motor task.

Novel fNIRS study on homogeneous symmetric feature-based transfer learning for brain-computer interface

The brain-computer interface (BCI) provides an alternate means of communication between the brain and external devices by recognizing the brain activities and translating them into external commands. The functional Near-Infrared Spectroscopy (fNIRS) is becoming popular as a non-invasive modality for brain activity detection. The recent trends show that deep learning has significantly enhanced the performance of the BCI systems. But the inherent bottleneck for deep learning (in the domain of BCI) is the requirement of the vast amount of training data, lengthy recalibrating time, and expensive computational resources for training deep networks. Building a high-quality, large-scale annotated dataset for deep learning-based BCI systems is exceptionally tedious, complex, and expensive. This study investigates the novel application of transfer learning for fNIRS-based BCI to solve three objective functions (concerns), i.e., the problem of insufficient training data, reduced training time, and increased accuracy. We applied symmetric homogeneous feature-based transfer learning on convolutional neural network (CNN) designed explicitly for fNIRS data collected from twenty-six (26) participants performing the n-back task. The results suggested that the proposed method achieves the maximum saturated accuracy sooner and outperformed the traditional CNN model on averaged accuracy by 25.58% in the exact duration of training time, reducing the training time, recalibrating time, and computational resources.

Novel Hybrid Brain-Computer Interface for Virtual Reality Applications Using Steady-State Visual-Evoked Potential-Based Brain-Computer Interface and Electrooculogram-Based Eye Tracking for Increased Information Transfer Rate

Brain-computer interfaces (BCIs) based on electroencephalogram (EEG) have recently attracted increasing attention in virtual reality (VR) applications as a promising tool for controlling virtual objects or generating commands in a "hands-free" manner. Video-oculography (VOG) has been frequently used as a tool to improve BCI performance by identifying the gaze location on the screen, however, current VOG devices are generally too expensive to be embedded in practical low-cost VR head-mounted display (HMD) systems. In this study, we proposed a novel calibration-free hybrid BCI system combining steady-state visual-evoked potential (SSVEP)-based BCI and electrooculogram (EOG)-based eye tracking to increase the information transfer rate (ITR) of a nine-target SSVEP-based BCI in VR environment. Experiments were repeated on three different frequency configurations of pattern-reversal checkerboard stimuli arranged in a 3 × 3 matrix. When a user was staring at one of the nine visual stimuli, the column containing the target stimulus was first identified based on the user's horizontal eye movement direction (left, middle, or right) classified using horizontal EOG recorded from a pair of electrodes that can be readily incorporated with any existing VR-HMD systems. Note that the EOG can be recorded using the same amplifier for recording SSVEP, unlike the VOG system. Then, the target visual stimulus was identified among the three visual stimuli vertically arranged in the selected column using the extension of multivariate synchronization index (EMSI) algorithm, one of the widely used SSVEP detection algorithms. In our experiments with 20 participants wearing a commercial VR-HMD system, it was shown that both the accuracy and ITR of the proposed hybrid BCI were significantly increased compared to those of the traditional SSVEP-based BCI in VR environment.

Steady-State Visual Evoked Potential-Based Brain–Computer Interface Using a Novel Visual Stimulus with Quick Response (QR) Code Pattern

Steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) systems suffer from low SSVEP response intensity and visual fatigue, resulting in lower accuracy when operating the system for continuous commands, such as an electric wheelchair control. This study proposes two SSVEP improvements to create a practical BCI for communication and control in disabled people. The first is flicker pattern modification for increasing SSVEP response through mixing (1) fundamental and first harmonic frequencies, and (2) two fundamental frequencies for an additional number of commands. The second method utilizes a quick response (QR) code for visual stimulus patterns to increase the SSVEP response and reduce visual fatigue. Eight different stimulus patterns from three flickering frequencies (7, 13, and 17 Hz) were presented to twelve participants for the test and score levels of visual fatigue. Two popular SSVEP methods, i.e., power spectral density (PSD) with Welch periodogram and canonical correlation analysis (CCA) with overlapping sliding window, are used to detect SSVEP intensity and response, compared to the checkerboard pattern. The results suggest that the QR code patterns can yield higher accuracy than checkerboard patterns for both PSD and CCA methods. Moreover, a QR code pattern with low frequency can reduce visual fatigue; however, visual fatigue can be easily affected by high flickering frequency. The findings can be used in the future to implement a real-time, SSVEP-based BCI for verifying user and system performance in actual environments.

Reducing calibration time in motor imagery-based BCIs by data alignment and empirical mode decomposition

One of the major reasons that limit the practical applications of a brain-computer interface (BCI) is its long calibration time. In this paper, we propose a novel approach to reducing the calibration time of motor imagery (MI)-based BCIs without sacrificing classification accuracy. The approach aims to augment the training set size of a new subject by generating artificial electroencephalogram (EEG) data from a few training trials initially available. The artificial EEG data are obtained by first performing empirical mode decomposition (EMD) and then mixing resulting intrinsic mode functions (IMFs). The original training trials are aligned to common reference point with Euclidean alignment (EA) method prior to EMD and pooled together with artificial trials as the expended training set, which is input into a linear discriminant analysis (LDA) classifier or a logistic regression (LR) classifier. The performance of the proposed algorithm is evaluated on two motor imagery (MI) data sets and compared with that of the algorithm trained with only real EEG data (Baseline) and the algorithm trained with expanded EEG data by EMD without data alignment. The experimental results showed that the proposed algorithm can significantly reduce the amount of training data needed to achieve a given performance level and thus is expected to facilitate the real-world applications of MI-based BCIs.

A Hybrid Brain-Computer Interface Based on Visual Evoked Potential and Pupillary Response

Brain-computer interface (BCI) based on steady-state visual evoked potential (SSVEP) has been widely studied due to the high information transfer rate (ITR), little user training, and wide subject applicability. However, there are also disadvantages such as visual discomfort and “BCI illiteracy.” To address these problems, this study proposes to use low-frequency stimulations (12 classes, 0.8–2.12 Hz with an interval of 0.12 Hz), which can simultaneously elicit visual evoked potential (VEP) and pupillary response (PR) to construct a hybrid BCI (h-BCI) system. Classification accuracy was calculated using supervised and unsupervised methods, respectively, and the hybrid accuracy was obtained using a decision fusion method to combine the information of VEP and PR. Online experimental results from 10 subjects showed that the averaged accuracy was 94.90 ± 2.34% (data length 1.5 s) for the supervised method and 91.88 ± 3.68% (data length 4 s) for the unsupervised method, which correspond to the ITR of 64.35 ± 3.07 bits/min (bpm) and 33.19 ± 2.38 bpm, respectively. Notably, the hybrid method achieved higher accuracy and ITR than that of VEP and PR for most subjects, especially for the short data length. Together with the subjects’ feedback on user experience, these results indicate that the proposed h-BCI with the low-frequency stimulation paradigm is more comfortable and favorable than the traditional SSVEP-BCI paradigm using the alpha frequency range.

Euler common spatial patterns for EEG classification

The technique of common spatial patterns (CSP) is a widely used method in the field of feature extraction of electroencephalogram (EEG) signals. Motivated by the fact that a cosine distance can enlarge the distance between samples of different classes, we propose the Euler CSP (e-CSP) for the feature extraction of EEG signals, and it is then used for EEG classification. The e-CSP is essentially the conventional CSP with the Euler representation. It includes the following two stages: each sample value is first mapped into a complex space by using the Euler representation, and then the conventional CSP is performed in the Euler space. Thus, the e-CSP is equivalent to applying the Euler representation as a kernel function to the input of the CSP. It is computationally as straightforward as the CSP. However, it extracts more discriminative features from the EEG signals. Extensive experimental results illustrate the discrimination ability of the e-CSP.

An Impending Paradigm Shift in Motor Imagery Based Brain-Computer Interfaces

The development of reliable assistive devices for patients that suffer from motor impairments following central nervous system lesions remains a major challenge in the field of non-invasive Brain-Computer Interfaces (BCIs). These approaches are predominated by electroencephalography and rely on advanced signal processing and machine learning methods to extract neural correlates of motor activity. However, despite tremendous and still ongoing efforts, their value as effective clinical tools remains limited. We advocate that a rather overlooked research avenue lies in efforts to question neurophysiological markers traditionally targeted in non-invasive motor BCIs. We propose an alternative approach grounded by recent fundamental advances in non-invasive neurophysiology, specifically subject-specific feature extraction of sensorimotor bursts of activity recorded via (possibly magnetoencephalography-optimized) electroencephalography. This path holds promise in overcoming a significant proportion of existing limitations, and could foster the wider adoption of online BCIs in rehabilitation protocols.

[Effectiveness of brain-computer interfaces and cognitive training using computer technologies in restoring cognitive functions in patients after stroke]

OBJECTIVE

To study the effectiveness of brain-computer interfaces (BCI) and cognitive training using computer technologies in restoring cognitive functions in poststroke patients.

MATERIAL AND METHODS

Thirty-four stroke patients (mean age 59.3±10.8 years) with stroke duration of 5.1±4.7 months, were included. To assess the effectiveness of treatment, patients before and after treatment were tested using memorization of words according to the method of Luria A.R. «10 words», the Montreal Cognitive Assessment Scale (MoCA), the Clock Drawing Test (CDT). All patients received standard rehabilitation therapy (exercise therapy, physiotherapy, sessions with a speech therapist-neuropsychologist). Patients of the first group additionally received training on the «Neurochat» complex, patients of the second group - on the «Exokist-2» complex, patients of the third group - cognitive training according to standard programs using computer technology and visual material.

RESULTS

Patients of the three groups showed a significant improvement in the total MoCA score: in the 1^st and 2^nd groups - p<0.01, in the 3^rd group - p<0.05. According to CDT, there was a significant change in the 2^nd group (p=0.018). The Luria method «10 words» revealed an improvement in memory in all groups (p<0.01, p<0.05), being more pronounced in the 1^st and 2^nd groups.

CONCLUSION

The effectiveness of BCI in restoring cognitive functions in patients after a stroke in comparison with cognitive training without BCI has been demonstrated. However, there are reasons to believe that various BCIs have a specific effect on cognitive functions and have their own target group.

Recent Advances and Current Trends in Brain-Computer Interface (BCI) Research and Their Applications

Brain-Computer Interface (BCI) provides direct communication between the brain and an external device. BCI systems have become a trendy field of research in recent years. These systems can be used in a variety of applications to help both disabled and healthy people. Concerning significant BCI progress, we may assume that these systems are not very far from real-world applications. This review has taken into account current trends in BCI research. In this survey, one hundred most cited articles from the WOS database were selected over the last four years. This survey is divided into several sectors. These sectors are Medicine, Communication and Control, Entertainment, and Other BCI applications. The application area, recording method, signal acquisition types, and countries of origin have been identified in each article. This survey provides an overview of the BCI articles published from 2016 to 2020 and their current trends and advances in different application areas.

An autonomous hybrid brain-computer interface system combined with eye-tracking in virtual environment

BACKGROUND

Brain-computer interface (BCI) has become an effective human-machine interactive way. However, the performance of the traditional BCI system needs to be further improved, such as flexibility, robustness, and accuracy. We aim to develop an autonomous hybrid BCI system combined with eye-tracking for the control tasks in the virtual environment.

NEW METHOD

This work developed an autonomous control strategy and proposed an effective fusion method for electroencephalogram (EEG) and eye tracking. For the autonomous control, the sliding window method was adopted to analyze the user's eye-gaze data. When the variance of eye-gaze data was less than the threshold, target recognition was triggered. EEG and eye-gaze data were synchronously collected and fused for classification. In addition, a fusion method based on particle swarm optimization (PSO) was proposed, which can find the best fusion weights to adapt to the differences of single modalities.

RESULTS

EEG data and eye-gaze data of 15 subjects in steady-state visual evoked potentials (SSVEP) tasks were collected to evaluate the effectiveness of the hybrid BCI system. The results showed that the PSO fusion method performed best in all fusion methods. And the proposed hybrid BCI system obtained higher accuracy and information transfer rate (ITR) than the single-modality.

COMPARISON WITH EXISTING METHODS

The PSO fusion method was compared with average weighting fusion, prior weighting fusion, support vector machine, decision tree, random forest, and extreme random tree.

CONCLUSION

The proposed methods of autonomous control and dual-modal fusion can improve the flexibility, robustness and classification performance of the hybrid BCI system.

Enhancing Sustained Attention

Arguably, automation is fast transforming many enterprise business processes, transforming operational jobs into monitoring tasks. Consequently, the ability to sustain attention during extended periods of monitoring is becoming a critical skill. This manuscript presents a Brain-Computer Interface (BCI) prototype which seeks to combat decrements in sustained attention during monitoring tasks within an enterprise system. A brain-computer interface is a system which uses physiological signals output by the user as an input. The goal is to better understand human responses while performing tasks involving decision and monitoring cycles, finding ways to improve performance and decrease on-task error. Decision readiness and the ability to synthesize complex and abundant information in a brief period during critical events has never been more important. Closed-loop control and motivational control theory were synthesized to provide the basis from which a framework for a prototype was developed to demonstrate the feasibility and value of a BCI in critical enterprise activities. In this pilot study, the BCI was implemented and evaluated through laboratory experimentation using an ecologically valid task. The results show that the technological artifact allowed users to regulate sustained attention positively while performing the task. Levels of sustained attention were shown to be higher in the conditions assisted by the BCI. Furthermore, this increased cognitive response seems to be related to increased on-task action and a small reduction in on-task errors. The research concludes with a discussion of the future research directions and their application in the enterprise.

Hybrid Systems to Boost EEG-Based Real-Time Action Decoding in Car Driving Scenarios

The complexity of concurrent cerebral processes underlying driving makes such human behavior one of the most studied real-world activities in neuroergonomics. Several attempts have been made to decode, both offline and online, cerebral activity during car driving with the ultimate goal to develop brain-based systems for assistive devices. Electroencephalography (EEG) is the cornerstone of these studies providing the highest temporal resolution to track those cerebral processes underlying overt behavior. Particularly when investigating real-world scenarios as driving, EEG is constrained by factors such as robustness, comfortability, and high data variability affecting the decoding performance. Hence, additional peripheral signals can be combined with EEG for increasing replicability and the overall performance of the brain-based action decoder. In this regard, hybrid systems have been proposed for the detection of braking and steering actions in driving scenarios to improve the predictive power of the single neurophysiological measurement. These recent results represent a proof of concept of the level of technological maturity. They may pave the way for increasing the predictive power of peripheral signals, such as electroculogram (EOG) and electromyography (EMG), collected in real-world scenarios when informed by EEG measurements, even if collected only offline in standard laboratory settings. The promising usability of such hybrid systems should be further investigated in other domains of neuroergonomics.

Classification of Individual Finger Movements from Right Hand Using fNIRS Signals

Functional near-infrared spectroscopy (fNIRS) is a comparatively new noninvasive, portable, and easy-to-use brain imaging modality. However, complicated dexterous tasks such as individual finger-tapping, particularly using one hand, have been not investigated using fNIRS technology. Twenty-four healthy volunteers participated in the individual finger-tapping experiment. Data were acquired from the motor cortex using sixteen sources and sixteen detectors. In this preliminary study, we applied standard fNIRS data processing pipeline, i.e., optical densities conversation, signal processing, feature extraction, and classification algorithm implementation. Physiological and non-physiological noise is removed using 4th order band-pass Butter-worth and 3rd order Savitzky-Golay filters. Eight spatial statistical features were selected: signal-mean, peak, minimum, Skewness, Kurtosis, variance, median, and peak-to-peak form data of oxygenated haemoglobin changes. Sophisticated machine learning algorithms were applied, such as support vector machine (SVM), random forests (RF), decision trees (DT), AdaBoost, quadratic discriminant analysis (QDA), Artificial neural networks (ANN), k-nearest neighbors (kNN), and extreme gradient boosting (XGBoost). The average classification accuracies achieved were 0.75±0.04, 0.75±0.05, and 0.77±0.06 using k-nearest neighbors (kNN), Random forest (RF) and XGBoost, respectively. KNN, RF and XGBoost classifiers performed exceptionally well on such a high-class problem. The results need to be further investigated. In the future, a more in-depth analysis of the signal in both temporal and spatial domains will be conducted to investigate the underlying facts. The accuracies achieved are promising results and could open up a new research direction leading to enrichment of control commands generation for fNIRS-based brain-computer interface applications.