A review of critical challenges in MI-BCI: From conventional to deep learning methods

Development of a humanoid robot control system based on AR-BCI and SLAM navigation

Brain-computer interface (BCI)-based robot combines BCI and robotics technology to realize the brain's intention to control the robot, which not only opens up a new way for the daily care of the disabled individuals, but also provides a new way of communication for normal people. However, the existing systems still have shortcomings in many aspects such as friendliness of human-computer interaction, and interaction efficient. This study developed a humanoid robot control system by integrating an augmented reality (AR)-based BCI with a simultaneous localization and mapping (SLAM)-based scheme for autonomous indoor navigation. An 8-target steady-state visual evoked potential (SSVEP)-based BCI was implemented to enable direct control of the humanoid robot by the user. A Microsoft HoloLens was utilized to display visual stimuli for eliciting SSVEPs. Filter bank canonical correlation analysis (FBCCA), a training-free method, was used to detect SSVEPs in this study. By leveraging SLAM technology, the proposed system alleviates the need for frequent control commands transmission from the user, thereby effectively reducing their workload. Online results from 12 healthy subjects showed this developed BCI system was able to select a command out of eight potential targets with an average accuracy of 94.79%. The autonomous navigation subsystem enabled the humanoid robot to autonomously navigate to a destination chosen utilizing the proposed BCI. Furthermore, all participants successfully completed the experimental task using the developed system without any prior training. These findings illustrate the feasibility of the developed system and its potential to contribute novel insights into humanoid robots control strategies.

Brain state and dynamic transition patterns of motor imagery revealed by the bayes hidden markov model

Motor imagery (MI) is a high-level cognitive process that has been widely applied to brain-computer inference (BCI) and motor recovery. In practical applications, however, huge individual differences and unclear neural mechanisms have seriously hindered the application of MI and BCI systems. Thus, it is urgently needed to explore MI from a new perspective. Here, we applied a hidden Markov model (HMM) to explore the dynamic organization patterns of left- and right-hand MI tasks. Eleven distinct HMM states were identified based on MI-related EEG data. We found that these states can be divided into three metastates by clustering analysis, showing a highly organized structure. We also assessed the probability activation of each HMM state across time. The results showed that the state probability activation of task-evoked have similar trends to that of event-related desynchronization/synchronization (ERD/ERS). By comparing the differences in temporal features of HMM states between left- and right-hand MI, we found notable variations in fractional occupancy, mean life time, mean interval time, and transition probability matrix across stages and states. Interestingly, we found that HMM states activated in the left occipital lobe had higher occupancy during the left-hand MI task, and conversely, during the right-hand MI task, HMM states activated in the right occipital lobe had higher occupancy. Moreover, significant correlations were observed between BCI performance and features of HMM states. Taken together, our findings explored dynamic networks underlying the MI-related process and provided a complementary understanding of different MI tasks, which may contribute to improving the MI-BCI systems.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-024-10099-9.

Ten quick tips for clinical electroencephalographic (EEG) data acquisition and signal processing

Electroencephalography (EEG) is a medical engineering technique aimed at recording the electric activity of the human brain. Brain signals derived from an EEG device can be processed and analyzed through computers by using digital signal processing, computational statistics, and machine learning techniques, that can lead to scientifically-relevant results and outcomes about how the brain works. In the last decades, the spread of EEG devices and the higher availability of EEG data, of computational resources, and of software packages for electroencephalography analysis has made EEG signal processing easier and faster to perform for any researcher worldwide. This increased ease to carry out computational analyses of EEG data, however, has made it easier to make mistakes, as well. And these mistakes, if unnoticed or treated wrongly, can in turn lead to wrong results or misleading outcomes, with worrisome consequences for patients and for the advancements of the knowledge about human brain. To tackle this problem, we present here our ten quick tips to perform electroencephalography signal processing analyses avoiding common mistakes: a short list of guidelines designed for beginners on what to do, how to do it, and what not to do when analyzing EEG data with a computer. We believe that following our quick recommendations can lead to better, more reliable and more robust results and outcome in clinical neuroscientific research.

Exploring the frontier: Transformer-based models in EEG signal analysis for brain-computer interfaces

This review systematically explores the application of transformer-based models in EEG signal processing and brain-computer interface (BCI) development, with a distinct focus on ensuring methodological rigour and adhering to empirical validations within the existing literature. By examining various transformer architectures, such as the Temporal Spatial Transformer Network (TSTN) and EEG Conformer, this review delineates their capabilities in mitigating challenges intrinsic to EEG data, such as noise and artifacts, and their subsequent implications on decoding and classification accuracies across disparate mental tasks. The analytical scope extends to a meticulous examination of attention mechanisms within transformer models, delineating their role in illuminating critical temporal and spatial EEG features and facilitating interpretability in model decision-making processes. The discourse additionally encapsulates emerging works that substantiate the efficacy of transformer models in noise reduction of EEG signals and diversifying applications beyond the conventional motor imagery paradigm. Furthermore, this review elucidates evident gaps and propounds exploratory avenues in the applications of pre-trained transformers in EEG analysis and the potential expansion into real-time and multi-task BCI applications. Collectively, this review distils extant knowledge, navigates through the empirical findings, and puts forward a structured synthesis, thereby serving as a conduit for informed future research endeavours in transformer-enhanced, EEG-based BCI systems.

Next-Gen Medical Imaging: U-Net Evolution and the Rise of Transformers

The advancement of medical imaging has profoundly impacted our understanding of the human body and various diseases. It has led to the continuous refinement of related technologies over many years. Despite these advancements, several challenges persist in the development of medical imaging, including data shortages characterized by low contrast, high noise levels, and limited image resolution. The U-Net architecture has significantly evolved to address these challenges, becoming a staple in medical imaging due to its effective performance and numerous updated versions. However, the emergence of Transformer-based models marks a new era in deep learning for medical imaging. These models and their variants promise substantial progress, necessitating a comparative analysis to comprehend recent advancements. This review begins by exploring the fundamental U-Net architecture and its variants, then examines the limitations encountered during its evolution. It then introduces the Transformer-based self-attention mechanism and investigates how modern models incorporate positional information. The review emphasizes the revolutionary potential of Transformer-based techniques, discusses their limitations, and outlines potential avenues for future research.

Optimizing motor imagery BCI models with hard trials removal and model refinement

Deep learning models have demonstrated remarkable performance in the classification of motor imagery BCI systems. However, these models exhibit sensitivity to challenging trials, often called hard trials, leading to performance degradation. In this paper, we address this issue by proposing two novel methods for identifying and mitigating the impact of hard trials on model performance. The first method leverages model prediction scores to discern hard trials. The second approach employs a quantitative explainable artificial intelligence (XAI) approach, enabling a more transparent and interpretable means of hard trial identification. The identified hard trials are removed from the entire motor imagery training and validation dataset, and the deep learning model is further re-trained using the dataset without hard trials. To evaluate the efficacy of these proposed methods, experiments were conducted on the Open BMI dataset. The results for hold-out analysis show that the proposed quantitative XAI- based hard trial removal method has statistically improved the average classification accuracy of the baseline deep CNN model from 63.77% to 68.70%, withp-value =7.66e-11for the subject-specific MI classification. Additionally, analyzing the scalp map representing the average relevance scores of correctly classified trials compared to a misclassified trial provides a deeper insight into identifying hard trials. The results indicate that the proposed quantitaive-based XAI approach outperformes the prediction-score-based approach in hard trial identification.

Three-stage transfer learning for motor imagery EEG recognition

Motor imagery (MI) paradigms have been widely used in neural rehabilitation and drowsiness state assessment. The progress in brain-computer interface (BCI) technology has emphasized the importance of accurately and efficiently detecting motor imagery intentions from electroencephalogram (EEG). Despite the recent breakthroughs made in developing EEG-based algorithms for decoding MI, the accuracy and efficiency of these models remain limited by technical challenges posed by cross-subject heterogeneity in EEG data processing and the scarcity of EEG data for training. Inspired by the optimal transport theory, this study aims to develop a novel three-stage transfer learning (TSTL) method, which uses the existing labeled data from a source domain to improve classification performance on an unlabeled target domain. Notably, the proposed method comprises three components, namely, the Riemannian tangent space mapping (RTSM), source domain transformer (SDT), and optimal subspace mapping (OSM). The RTSM maps a symmetric positive definite matrix from the Riemannian space to the tangent space to minimize the marginal probability distribution drift. The SDT transforms the source domain to a target domain by finding the optimal transport mapping matrix to reduce the joint probability distribution differences. The OSM finally maps the transformed source domain and original target domain to the same subspace to further mitigate the distribution discrepancy. The performance of the proposed method was validated on two public BCI datasets, and the average accuracy of the algorithm on two datasets was 72.24% and 69.29%. Our results demonstrated the improved performance of EEG-based MI detection in comparison with state-of-the-art algorithms.

A robust multi-branch multi-attention-mechanism EEGNet for motor imagery BCI decoding

BACKGROUND

Motor-Imagery-based Brain-Computer Interface (MI-BCI) is a promising technology to assist communication, movement, and neurological rehabilitation for motor-impaired individuals. Electroencephalography (EEG) decoding techniques using deep learning (DL) possess noteworthy advantages due to automatic feature extraction and end-to-end learning. However, the DL-based EEG decoding models tend to show large variations due to intersubject variability of EEG, which results from inconsistencies of different subjects' optimal hyperparameters.

NEW METHODS

This study proposes a multi-branch multi-attention mechanism EEGNet model (MBMANet) for robust decoding. It applies the multi-branch EEGNet structure to achieve various feature extractions. Further, the different attention mechanisms introduced in each branch attain diverse adaptive weight adjustments. This combination of multi-branch and multi-attention mechanisms allows for multi-level feature fusion to provide robust decoding for different subjects.

RESULTS

The MBMANet model has a four-classification accuracy of 83.18% and kappa of 0.776 on the BCI Competition IV-2a dataset, which outperforms other eight CNN-based decoding models. This consistently satisfactory performance across all nine subjects indicates that the proposed model is robust.

CONCLUSIONS

The combine of multi-branch and multi-attention mechanisms empowers the DL-based models to adaptively learn different EEG features, which provides a feasible solution for dealing with data variability. It also gives the MBMANet model more accurate decoding of motion intentions and lower training costs, thus improving the MI-BCI's utility and robustness.

A Data Augmentation Method for Motor Imagery EEG Signals Based on DCGAN-GP Network

Motor imagery electroencephalography (EEG) signals have garnered attention in brain-computer interface (BCI) research due to their potential in promoting motor rehabilitation and control. However, the limited availability of labeled data poses challenges for training robust classifiers. In this study, we propose a novel data augmentation method utilizing an improved Deep Convolutional Generative Adversarial Network with Gradient Penalty (DCGAN-GP) to address this issue. We transformed raw EEG signals into two-dimensional time-frequency maps and employed a DCGAN-GP network to generate synthetic time-frequency representations resembling real data. Validation experiments were conducted on the BCI IV 2b dataset, comparing the performance of classifiers trained with augmented and unaugmented data. Results demonstrated that classifiers trained with synthetic data exhibit enhanced robustness across multiple subjects and achieve higher classification accuracy. Our findings highlight the effectiveness of utilizing a DCGAN-GP-generated synthetic EEG data to improve classifier performance in distinguishing different motor imagery tasks. Thus, the proposed data augmentation method based on a DCGAN-GP offers a promising avenue for enhancing BCI system performance, overcoming data scarcity challenges, and bolstering classifier robustness, thereby providing substantial support for the broader adoption of BCI technology in real-world applications.

Brain-machine interface based on deep learning to control asynchronously a lower-limb robotic exoskeleton: a case-of-study

BACKGROUND

This research focused on the development of a motor imagery (MI) based brain-machine interface (BMI) using deep learning algorithms to control a lower-limb robotic exoskeleton. The study aimed to overcome the limitations of traditional BMI approaches by leveraging the advantages of deep learning, such as automated feature extraction and transfer learning. The experimental protocol to evaluate the BMI was designed as asynchronous, allowing subjects to perform mental tasks at their own will.

METHODS

A total of five healthy able-bodied subjects were enrolled in this study to participate in a series of experimental sessions. The brain signals from two of these sessions were used to develop a generic deep learning model through transfer learning. Subsequently, this model was fine-tuned during the remaining sessions and subjected to evaluation. Three distinct deep learning approaches were compared: one that did not undergo fine-tuning, another that fine-tuned all layers of the model, and a third one that fine-tuned only the last three layers. The evaluation phase involved the exclusive closed-loop control of the exoskeleton device by the participants' neural activity using the second deep learning approach for the decoding.

RESULTS

The three deep learning approaches were assessed in comparison to an approach based on spatial features that was trained for each subject and experimental session, demonstrating their superior performance. Interestingly, the deep learning approach without fine-tuning achieved comparable performance to the features-based approach, indicating that a generic model trained on data from different individuals and previous sessions can yield similar efficacy. Among the three deep learning approaches compared, fine-tuning all layer weights demonstrated the highest performance.

CONCLUSION

This research represents an initial stride toward future calibration-free methods. Despite the efforts to diminish calibration time by leveraging data from other subjects, complete elimination proved unattainable. The study's discoveries hold notable significance for advancing calibration-free approaches, offering the promise of minimizing the need for training trials. Furthermore, the experimental evaluation protocol employed in this study aimed to replicate real-life scenarios, granting participants a higher degree of autonomy in decision-making regarding actions such as walking or stopping gait.

Brain-computer interface digital prescription for neurological disorders

Neurological and psychiatric diseases can lead to motor, language, emotional disorder, and cognitive, hearing or visual impairment By decoding the intention of the brain in real time, the Brain-computer interface (BCI) can first assist in the diagnosis of diseases, and can also compensate for its damaged function by directly interacting with the environment; In addition, provide output signals in various forms, such as actual motion, tactile or visual feedback, to assist in rehabilitation training; Further intervention in brain disorders is achieved by close-looped neural modulation. In this article, we envision the future BCI digital prescription system for patients with different functional disorders and discuss the key contents in the prescription the brain signals, coding and decoding protocols and interaction paradigms, and assistive technology. Then, we discuss the details that need to be specially included in the digital prescription for different intervention technologies. The third part summarizes previous examples of intervention, focusing on how to select appropriate interaction paradigms for patients with different functional impairments. For the last part, we discussed the indicators and influencing factors in evaluating the therapeutic effect of BCI as intervention.

Multiclass classification of motor imagery tasks based on multi-branch convolutional neural network and temporal convolutional network model

Motor imagery (MI) is a cognitive process wherein an individual mentally rehearses a specific movement without physically executing it. Recently, MI-based brain-computer interface (BCI) has attracted widespread attention. However, accurate decoding of MI and understanding of neural mechanisms still face huge challenges. These seriously hinder the clinical application and development of BCI systems based on MI. Thus, it is very necessary to develop new methods to decode MI tasks. In this work, we propose a multi-branch convolutional neural network (MBCNN) with a temporal convolutional network (TCN), an end-to-end deep learning framework to decode multi-class MI tasks. We first used MBCNN to capture the MI electroencephalography signals information on temporal and spectral domains through different convolutional kernels. Then, we introduce TCN to extract more discriminative features. The within-subject cross-session strategy is used to validate the classification performance on the dataset of BCI Competition IV-2a. The results showed that we achieved 75.08% average accuracy for 4-class MI task classification, outperforming several state-of-the-art approaches. The proposed MBCNN-TCN-Net framework successfully captures discriminative features and decodes MI tasks effectively, improving the performance of MI-BCIs. Our findings could provide significant potential for improving the clinical application and development of MI-based BCI systems.

Non-destructive optical sensing technologies for advancing the egg industry toward Industry 4.0: A review

The egg is considered one of the best sources of dietary protein, and has an important role in human growth and development. With the increase in the world's population, per capita egg consumption is also increasing. Ground-breaking technological developments have led to numerous inventions like the Internet of Things (IoT), various optical sensors, robotics, artificial intelligence (AI), big data, and cloud computing, transforming the conventional industry into a smart and sustainable egg industry, also known as Egg Industry 4.0 (EI 4.0). The EI 4.0 concept has the potential to improve automation, enhance biosecurity, promote the safeguarding of animal welfare, increase intelligent grading and quality inspection, and increase efficiency. For a sustainable Industry 4.0 transformation, it is important to analyze available technologies, the latest research, existing limitations, and prospects. This review examines the existing non-destructive optical sensing technologies for the egg industry. It provides information and insights on the different components of EI 4.0, including emerging EI 4.0 technologies for egg production, quality inspection, and grading. Furthermore, drawbacks of current EI 4.0 technologies, potential workarounds, and future trends were critically analyzed. This review can help policymakers, industrialists, and academicians to better understand the integration of non-destructive technologies and automation. This integration has the potential to increase productivity, improve quality control, and optimize resource management toward sustainable development of the egg industry.

BrainWave-Scattering Net: a lightweight network for EEG-based motor imagery recognition

Objective.Brain-computer interfaces (BCIs) enable a direct communication of the brain with the external world, using one's neural activity, measured by electroencephalography (EEG) signals. In recent years, convolutional neural networks (CNNs) have been widely used to perform automatic feature extraction and classification in various EEG-based tasks. However, their undeniable benefits are counterbalanced by the lack of interpretability properties as well as the inability to perform sufficiently when only limited amount of training data is available.Approach.In this work, we introduce a novel, lightweight, fully-learnable neural network architecture that relies on Gabor filters to delocalize EEG signal information into scattering decomposition paths along frequency and slow-varying temporal modulations.Main results.We utilize our network in two distinct modeling settings, for building either a generic (training across subjects) or a personalized (training within a subject) classifier.Significance.In both cases, using two different publicly available datasets and one in-house collected dataset, we demonstrate high performance for our model with considerably less number of trainable parameters as well as shorter training time compared to other state-of-the-art deep architectures. Moreover, our network demonstrates enhanced interpretability properties emerging at the level of the temporal filtering operation and enables us to train efficient personalized BCI models with limited amount of training data.