CNN-based classification of fNIRS signals in motor imagery BCI system

Simultaneous EEG-fNIRS Data Classification Through Selective Channel Representation and Spectrogram Imaging

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) can facilitate the advancement of brain-computer interfaces (BCIs). However, existing research in this domain has grappled with the challenge of the efficient selection of features, resulting in the underutilization of the temporal richness of EEG and the spatial specificity of fNIRS data.To effectively address this challenge, this study proposed a deep learning architecture called the multimodal DenseNet fusion (MDNF) model that was trained on two-dimensional (2D) EEG data images, leveraging advanced feature extraction techniques. The model transformed EEG data into 2D images using a short-time Fourier transform, applied transfer learning to extract discriminative features, and consequently integrated them with fNIRS-derived spectral entropy features. This approach aimed to bridge existing gaps in EEG-fNIRS-based BCI research by enhancing classification accuracy and versatility across various cognitive and motor imagery tasks.Experimental results on two public datasets demonstrated the superiority of our model over existing state-of-the-art methods.Thus, the high accuracy and precise feature utilization of the MDNF model demonstrates the potential in clinical applications for neurodiagnostics and rehabilitation, thereby paving the method for patient-specific therapeutic strategies.

Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks

The detection of the cognitive tasks performed by a subject during data acquisition of a neuroimaging method has a wide range of applications: functioning of brain-computer interface (BCI), detection of neuronal disorders, neurorehabilitation for disabled patients, and many others. Recent studies show that the combination or fusion of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) demonstrates improved classification and detection performance compared to sole-EEG and sole-fNIRS. Deep learning (DL) networks are suitable for the classification of large volume time-series data like EEG and fNIRS. This study performs the decision fusion of EEG and fNIRS. The classification of EEG, fNIRS, and decision-fused EEG-fNIRSinto cognitive task labels is performed by DL networks. Two different open-source datasets of simultaneously recorded EEG and fNIRS are examined in this study. Dataset 01 is comprised of 26 subjects performing 3 cognitive tasks: n-back, discrimination or selection response (DSR), and word generation (WG). After data acquisition, fNIRS is converted to oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) in Dataset 01. Dataset 02 is comprised of 29 subjects who performed 2 tasks: motor imagery and mental arithmetic. The classification procedure of EEG and fNIRS (or HbO2, HbR) are carried out by 7 DL classifiers: convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), CNN-LSTM, CNN-GRU, LSTM-GRU, and CNN-LSTM-GRU. After the classification of single modalities, their prediction scores or decisions are combined to obtain the decision-fused modality. The classification performance is measured by overall accuracy and area under the ROC curve (AUC). The highest accuracy and AUC recorded in Dataset 01 are 96% and 100% respectively; both by the decision fusion modality using CNN-LSTM-GRU. For Dataset 02, the highest accuracy and AUC are 82.76% and 90.44% respectively; both by the decision fusion modality using CNN-LSTM. The experimental result shows that decision-fused EEG-HbO2-HbR and EEG-fNIRSdeliver higher performances compared to their constituent unimodalities in most cases. For DL classifiers, CNN-LSTM-GRU in Dataset 01 and CNN-LSTM in Dataset 02 yield the highest performance.

Working memory load recognition with deep learning time series classification

Working memory load (WML) is one of the widely applied signals in the areas of human-machine interaction. The precise evaluation of the WML is crucial for this kind of application. This study aims to propose a deep learning (DL) time series classification (TSC) model for inter-subject WML decoding. We used fNIRS to record the hemodynamic signals of 27 participants during visual working memory tasks. Traditional machine learning and deep time series classification algorithms were respectively used for intra-subject and inter-subject WML decoding from the collected blood oxygen signals. The intra-subject classification accuracy of LDA and SVM were 94.6% and 79.1%. Our proposed TAResnet-BiLSTM model had the highest inter-subject WML decoding accuracy, reaching 92.4%. This study provides a new idea and method for the brain-computer interface application of fNIRS in real-time WML detection.

fNIRS-EEG BCIs for Motor Rehabilitation: A Review

Motor impairment has a profound impact on a significant number of individuals, leading to a substantial demand for rehabilitation services. Through brain-computer interfaces (BCIs), people with severe motor disabilities could have improved communication with others and control appropriately designed robotic prosthetics, so as to (at least partially) restore their motor abilities. BCI plays a pivotal role in promoting smoother communication and interactions between individuals with motor impairments and others. Moreover, they enable the direct control of assistive devices through brain signals. In particular, their most significant potential lies in the realm of motor rehabilitation, where BCIs can offer real-time feedback to assist users in their training and continuously monitor the brain's state throughout the entire rehabilitation process. Hybridization of different brain-sensing modalities, especially functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), has shown great potential in the creation of BCIs for rehabilitating the motor-impaired populations. EEG, as a well-established methodology, can be combined with fNIRS to compensate for the inherent disadvantages and achieve higher temporal and spatial resolution. This paper reviews the recent works in hybrid fNIRS-EEG BCIs for motor rehabilitation, emphasizing the methodologies that utilized motor imagery. An overview of the BCI system and its key components was introduced, followed by an introduction to various devices, strengths and weaknesses of different signal processing techniques, and applications in neuroscience and clinical contexts. The review concludes by discussing the possible challenges and opportunities for future development.

Research on Emotion Recognition Method of Cerebral Blood Oxygen Signal Based on CNN-Transformer Network

In recent years, research on emotion recognition has become more and more popular, but there are few studies on emotion recognition based on cerebral blood oxygen signals. Since the electroencephalogram (EEG) is easily disturbed by eye movement and the portability is not high, this study uses a more comfortable and convenient functional near-infrared spectroscopy (fNIRS) system to record brain signals from participants while watching three different types of video clips. During the experiment, the changes in cerebral blood oxygen concentration in the 8 channels of the prefrontal cortex of the brain were collected and analyzed. We processed and divided the collected cerebral blood oxygen data, and used multiple classifiers to realize the identification of the three emotional states of joy, neutrality, and sadness. Since the classification accuracy of the convolutional neural network (CNN) in this research is not significantly superior to that of the XGBoost algorithm, this paper proposes a CNN-Transformer network based on the characteristics of time series data to improve the classification accuracy of ternary emotions. The network first uses convolution operations to extract channel features from multi-channel time series, then the features and the output information of the fully connected layer are input to the Transformer netork structure, and its multi-head attention mechanism is used to focus on different channel domain information, which has better spatiality. The experimental results show that the CNN-Transformer network can achieve 86.7% classification accuracy for ternary emotions, which is about 5% higher than the accuracy of CNN, and this provides some help for other research in the field of emotion recognition based on time series data such as fNIRS.

Consumer-Grade Electroencephalogram and Functional Near-Infrared Spectroscopy Neurofeedback Technologies for Mental Health and Wellbeing

Neurofeedback, utilizing an electroencephalogram (EEG) and/or a functional near-infrared spectroscopy (fNIRS) device, is a real-time measurement of brain activity directed toward controlling and optimizing brain function. This treatment has often been attributed to improvements in disorders such as ADHD, anxiety, depression, and epilepsy, among others. While there is evidence suggesting the efficacy of neurofeedback devices, the research is still inconclusive. The applicability of the measurements and parameters of consumer neurofeedback wearable devices has improved, but the literature on measurement techniques lacks rigorously controlled trials. This paper presents a survey and literary review of consumer neurofeedback devices and the direction toward clinical applications and diagnoses. Relevant devices are highlighted and compared for treatment parameters, structural composition, available software, and clinical appeal. Finally, a conclusion on future applications of these systems is discussed through the comparison of their advantages and drawbacks.

CGAN-rIRN: a data-augmented deep learning approach to accurate classification of mental tasks for a fNIRS-based brain-computer interface

Functional near-infrared spectroscopy (fNIRS) is increasingly used to investigate different mental tasks for brain-computer interface (BCI) control due to its excellent environmental and motion robustness. Feature extraction and classification strategy for fNIRS signal are essential to enhance the classification accuracy of voluntarily controlled BCI systems. The limitation of traditional machine learning classifiers (MLCs) lies in manual feature engineering, which is considered as one of the drawbacks that reduce accuracy. Since the fNIRS signal is a typical multivariate time series with multi-dimensionality and complexity, it makes the deep learning classifier (DLC) ideal for classifying neural activation patterns. However, the inherent bottleneck of DLCs is the requirement of substantial-scale, high-quality labeled training data and expensive computational resources to train deep networks. The existing DLCs for classifying mental tasks do not fully consider the temporal and spatial properties of fNIRS signals. Therefore, a specifically-designed DLC is desired to classify multi-tasks with high accuracy in fNIRS-BCI. To this end, we herein propose a novel data-augmented DLC to accurately classify mental tasks, which employs a convolution-based conditional generative adversarial network (CGAN) for data augmentation and a revised Inception-ResNet (rIRN) based DLC. The CGAN is utilized to generate class-specific synthetic fNIRS signals to augment the training dataset. The network architecture of rIRN is elaborately designed in accordance with the characteristics of the fNIRS signal, with serial multiple spatial and temporal feature extraction modules (FEMs), where each FEM performs deep and multi-scale feature extraction and fusion. The results of the paradigm experiments show that the proposed CGAN-rIRN approach improves the single-trial accuracy for mental arithmetic and mental singing tasks in both the data augmentation and classifier, as compared to the traditional MLCs and the commonly used DLCs. The proposed fully data-driven hybrid deep learning approach paves a promising way to improve the classification performance of volitional control fNIRS-BCI.

System Derived Spatial-Temporal CNN for High-Density fNIRS BCI

An intuitive and generalisable approach to spatial-temporal feature extraction for high-density (HD) functional Near-Infrared Spectroscopy (fNIRS) brain-computer interface (BCI) is proposed, demonstrated here using Frequency-Domain (FD) fNIRS for motor-task classification. Enabled by the HD probe design, layered topographical maps of Oxy/deOxy Haemoglobin changes are used to train a 3D convolutional neural network (CNN), enabling simultaneous extraction of spatial and temporal features. The proposed spatial-temporal CNN is shown to effectively exploit the spatial relationships in HD fNIRS measurements to improve the classification of the functional haemodynamic response, achieving an average F1 score of 0.69 across seven subjects in a mixed subjects training scheme, and improving subject-independent classification as compared to a standard temporal CNN.

Combining robust level extraction and unsupervised adaptive classification for high-accuracy fNIRS-BCI: An evidence on single-trial differentiation between mentally arithmetic- and singing-tasks

Functional near-infrared spectroscopy (fNIRS) is a safe and non-invasive optical imaging technique that is being increasingly used in brain-computer interfaces (BCIs) to recognize mental tasks. Unlike electroencephalography (EEG) which directly measures neural activation, fNIRS signals reflect neurovascular-coupling inducing hemodynamic response that can be slow in time and varying in the pattern. The established classifiers extend the EEG-ones by mostly employing the feature based supervised models such as the support vector machine (SVM) and linear discriminant analysis (LDA), and fail to timely characterize the level-sensitive hemodynamic pattern. A dedicated classifier is desired for intentional activity recognition of fNIRS-BCI, including the adaptive acquisition of response relevant features and accurate discrimination of implied ideas. To this end, we herein propose a specifically-designed joint adaptive classification method that combines a Kalman filtering (KF) for robust level extraction and an adaptive Gaussian mixture model (a-GMM) for enhanced pattern recognition. The simulative investigations and paradigm experiments have shown that the proposed KF/a-GMM classification method can effectively track the random variations of task-evoked brain activation patterns, and improve the accuracy of single-trial classification task of mental arithmetic vs. mental singing, as compared to the conventional methods, e.g., those that employ combinations of the band-pass filtering (BPF) based feature extractors (mean, slope, and variance, etc.) and the classical recognizers (GMM, SVM, and LDA). The proposed approach paves a promising way for developing the real-time fNIRS-BCI technique.

Deep neural network to differentiate brain activity between patients with euthymic bipolar disorders and healthy controls during verbal fluency performance: A multichannel near-infrared spectroscopy study

In this study, we aimed to differentiate between euthymic bipolar disorder (BD) patients and healthy controls (HC) based on frontal activity measured by fNIRS that were converted to spectrograms with Convolutional Neural Networks (CNN). And also, we investigated brain regions that cause this distinction. In total, 29 BD patients and 28 HCs were recruited. Their brain cortical activities were measured using fNIRS while performing letter versions of VFT. Each one of the 24 fNIRS channels was converted to a 2D spectrogram on which a CNN architecture was designed and utilized for classification. We found that our CNN algorithm using fNIRS activity during a VFT is able to differentiate subjects with BD from healthy controls with 90% accuracy, 80% sensitivity, and 100% specificity. Moreover, validation performance reached an AUC of 94%. From our individual channel analyses, we observed channels corresponding to the left inferior frontal gyrus (left-IFC), medial frontal cortex (MFC), right dorsolateral prefrontal cortex (DLPFC), Broca area, and right premotor have considerable activity variation to distinguish patients from HC. fNIRS activity during VFT can be used as a potential marker to classify euthymic BD patients from HCs. Activity particularly in the MFC, left-IFC, Broca's area, and DLPFC have a considerable variation to distinguish patients from healthy controls.

Deep learning in fNIRS: a review

Significance: Optical neuroimaging has become a well-established clinical and research tool to monitor cortical activations in the human brain. It is notable that outcomes of functional near-infrared spectroscopy (fNIRS) studies depend heavily on the data processing pipeline and classification model employed. Recently, deep learning (DL) methodologies have demonstrated fast and accurate performances in data processing and classification tasks across many biomedical fields. Aim: We aim to review the emerging DL applications in fNIRS studies. Approach: We first introduce some of the commonly used DL techniques. Then, the review summarizes current DL work in some of the most active areas of this field, including brain-computer interface, neuro-impairment diagnosis, and neuroscience discovery. Results: Of the 63 papers considered in this review, 32 report a comparative study of DL techniques to traditional machine learning techniques where 26 have been shown outperforming the latter in terms of the classification accuracy. In addition, eight studies also utilize DL to reduce the amount of preprocessing typically done with fNIRS data or increase the amount of data via data augmentation. Conclusions: The application of DL techniques to fNIRS studies has shown to mitigate many of the hurdles present in fNIRS studies such as lengthy data preprocessing or small sample sizes while achieving comparable or improved classification accuracy.

A General and Scalable Vision Framework for Functional Near-Infrared Spectroscopy Classification

Functional near-infrared spectroscopy (fNIRS), a non-invasive optical technique, is widely used to monitor brain activities for disease diagnosis and brain-computer interfaces (BCIs). Deep learning-based fNIRS classification faces three major barriers: limited datasets, confusing evaluation criteria, and domain barriers. We apply more appropriate evaluation methods to three open-access datasets to solve the first two barriers. For domain barriers, we propose a general and scalable vision fNIRS framework that converts multi-channel fNIRS signals into multi-channel virtual images using the Gramian angular difference field (GADF). We use the framework to train state-of-the-art visual models from computer vision (CV) within a few minutes, and the classification performance is competitive with the latest fNIRS models. In cross-validation experiments, visual models achieve the highest average classification results of 78.68% and 73.92% on mental arithmetic and word generation tasks, respectively. Although visual models are slightly lower than the fNIRS models on unilateral finger- and foot-tapping tasks, the F1-score and kappa coefficient indicate that these differences are insignificant in subject-independent experiments. Furthermore, we study fNIRS signal representations and the classification performance of sequence-to-image methods. We hope to introduce rich achievements from the CV domain to improve fNIRS classification research.

Decoding Brain Signals to Classify Gait Direction Anticipation

The use of brain-computer interface (BCI) technology has emerged as a promising rehabilitation approach for patients with motor function and motor-related disorders. BCIs provide an augmentative communication platform for controlling advanced assistive robots such as a lower-limb exoskeleton. Brain recordings collected by an electroencephalography (EEG) system have been employed in the BCI platform to command the exoskeleton. To date, the literature on this topic is limited to the prediction of gait intention and gait variations from EEG signals. This study, however, aims to predict the anticipated gait direction using a stream of EEG signals collected from the brain cortex. Three healthy participants (age range: 29-31, 2 female) were recruited. While wearing the EEG device, the participants were instructed to initiate gait movement toward the direction of the arrow triggers (pointing forward, backward, left, or right) being shown on a screen with a blank white background. Collected EEG data was then epoched around the trigger timepoints. These epochs were then converted to the time-frequency domain using event- related synchronization (ERS) and event-related desynchronization (ERD) methods. Finally, the classification pipeline was constructed using logistic regression (LR), support vector machine (SVM), and convolutional neural network (CNN). A ten-fold cross-validation scheme was used to evaluate the classification performance. The results revealed that the CNN classifier outperforms the other two classifiers with an accuracy of 0.75. Clinical Relevance - The outcome of this study has the potential to be ultimately used for interactive navigation of the lower-limb exoskeletons during robotic rehabilitation therapy and enhance neurodegeneration and neuroplasticity in a wide range of individuals with lower-limb motor function disabilities.

Discriminative Frequencies and Temporal EEG Segmentation in the Motor Imagery Classification Approach

A linear discriminant analysis transformation-based approach to the classification of three different motor imagery types for brain–computer interfaces was considered. The study involved 16 conditionally healthy subjects (12 men, 4 women, mean age of 21.5 years). First, the search for subject-specific discriminative frequencies was conducted in the task of movement-related activity. This procedure was shown to increase the classification accuracy compared to the conditional common spatial pattern (CSP) algorithm, followed by a linear classifier considered as a baseline approach. In addition, an original approach to finding discriminative temporal segments for each motor imagery was tested. This led to a further increase in accuracy under the conditions of using Hjorth parameters and interchannel correlation coefficients as features calculated for the EEG segments. In particular, classification by the latter feature led to the best accuracy of 71.6%, averaged over all subjects (intrasubject classification), and, surprisingly, it also allowed us to obtain a comparable value of intersubject classification accuracy of 68%. Furthermore, scatter plots demonstrated that two out of three pairs of motor imagery were discriminated by the approach presented.

Analyzing Classification Performance of fNIRS-BCI for Gait Rehabilitation Using Deep Neural Networks

This research presents a brain-computer interface (BCI) framework for brain signal classification using deep learning (DL) and machine learning (ML) approaches on functional near-infrared spectroscopy (fNIRS) signals. fNIRS signals of motor execution for walking and rest tasks are acquired from the primary motor cortex in the brain's left hemisphere for nine subjects. DL algorithms, including convolutional neural networks (CNNs), long short-term memory (LSTM), and bidirectional LSTM (Bi-LSTM) are used to achieve average classification accuracies of 88.50%, 84.24%, and 85.13%, respectively. For comparison purposes, three conventional ML algorithms, support vector machine (SVM), k-nearest neighbor (k-NN), and linear discriminant analysis (LDA) are also used for classification, resulting in average classification accuracies of 73.91%, 74.24%, and 65.85%, respectively. This study successfully demonstrates that the enhanced performance of fNIRS-BCI can be achieved in terms of classification accuracy using DL approaches compared to conventional ML approaches. Furthermore, the control commands generated by these classifiers can be used to initiate and stop the gait cycle of the lower limb exoskeleton for gait rehabilitation.

Feature Selection Using Extreme Gradient Boosting Bayesian Optimization to upgrade the Classification Performance of Motor Imagery signals for BCI

BACKGROUND

A motor imagery (MI) based brain computer interface (BCI) is a challenging nonmuscular connection system used to independently perform movement-related tasks. It is gaining increasing prominence in helping paralyzed individuals communicate with the real world. Achieving better classification accuracy is the major concern in the field of motor imagery-based BCI. To upgrade the classification performance, relevant features play a vital role. The relevant features can be selected by the extreme gradient Bayesian optimization (XGBO) method.

METHODS

In this paper, a combination of time-, frequency-, and spatial-related MI features are extracted to design a reliable MI-BCI system. The proposed method incorporates the XGBO algorithm for feature selection and the random forest for the classification of EEG signals. The potency of the proposed system was investigated using two public EEG datasets (BCI Competition III dataset IIIa and dataset IVa). A novel XGBO algorithm increases the accuracy and reduces the time consumption by reducing the dimensionality of features. The proposed algorithm selects the minimum number of features that increase the computational efficacy for MI-based BCI systems.

COMPARISON WITH EXISTING METHODS

The proposed method is compared with ANOVA, sequential forward selection, recursive feature elimination, and LASSO methods and the accuracy rate is increased with the lowest computation time.

RESULTS

The proposed method achieves mean accuracies of 94.44% and 88.72% and classification errors of 5.56% and 11.28% for Datasets IIIa and IVa, respectively. It outperforms four state-of-art methods with 0.87% and 0.59% increases in the accuracy for Datasets IIIa and IVa, respectively.

Research on fNIRS Recognition Method of Upper Limb Movement Intention

This paper aims at realizing upper limb rehabilitation training by using an fNIRS-BCI system. This article mainly focuses on the analysis and research of the cerebral blood oxygen signal in the system, and gradually extends the analysis and recognition method of the movement intention in the cerebral blood oxygen signal to the actual brain-computer interface system. Fifty subjects completed four upper limb movement paradigms: Lifting-up, putting down, pulling back, and pushing forward. Then, their near-infrared data and movement trigger signals were collected. In terms of the recognition algorithm for detecting the initial intention of upper limb movements, gradient boosting tree (GBDT) and random forest (RF) were selected for classification experiments. Finally, RF classifier with better comprehensive indicators was selected as the final classification algorithm. The best offline recognition rate was 94.4% (151/160). The ReliefF algorithm based on distance measurement and the genetic algorithm proposed in the genetic theory were used to select features. In terms of upper limb motion state recognition algorithms, logistic regression (LR), support vector machine (SVM), naive Bayes (NB), and linear discriminant analysis (LDA) were selected for experiments. Kappa coefficient was used as the classification index to evaluate the performance of the classifier. Finally, SVM classification got the best performance, and the four-class recognition accuracy rate was 84.4%. The results show that RF and SVM can achieve high recognition accuracy in motion intentions and the upper limb rehabilitation system designed in this paper has great application significance.