Enhanced Accuracy for Multiclass Mental Workload Detection Using Long Short-Term Memory for Brain-Computer Interface

Cognitive workload estimation using physiological measures: a review

Estimating cognitive workload levels is an emerging research topic in the cognitive neuroscience domain, as participants' performance is highly influenced by cognitive overload or underload results. Different physiological measures such as Electroencephalography (EEG), Functional Magnetic Resonance Imaging, Functional near-infrared spectroscopy, respiratory activity, and eye activity are efficiently used to estimate workload levels with the help of machine learning or deep learning techniques. Some reviews focus only on EEG-based workload estimation using machine learning classifiers or multimodal fusion of different physiological measures for workload estimation. However, a detailed analysis of all physiological measures for estimating cognitive workload levels still needs to be discovered. Thus, this survey highlights the in-depth analysis of all the physiological measures for assessing cognitive workload. This survey emphasizes the basics of cognitive workload, open-access datasets, the experimental paradigm of cognitive tasks, and different measures for estimating workload levels. Lastly, we emphasize the significant findings from this review and identify the open challenges. In addition, we also specify future scopes for researchers to overcome those challenges.

Decoding lower-limb kinematic parameters during pedaling tasks using deep learning approaches and EEG

Stroke is a neurological condition that usually results in the loss of voluntary control of body movements, making it difficult for individuals to perform activities of daily living (ADLs). Brain-computer interfaces (BCIs) integrated into robotic systems, such as motorized mini exercise bikes (MMEBs), have been demonstrated to be suitable for restoring gait-related functions. However, kinematic estimation of continuous motion in BCI systems based on electroencephalography (EEG) remains a challenge for the scientific community. This study proposes a comparative analysis to evaluate two artificial neural network (ANN)-based decoders to estimate three lower-limb kinematic parameters: x- and y-axis position of the ankle and knee joint angle during pedaling tasks. Long short-term memory (LSTM) was used as a recurrent neural network (RNN), which reached Pearson correlation coefficient (PCC) scores close to 0.58 by reconstructing kinematic parameters from the EEG features on the delta band using a time window of 250 ms. These estimates were evaluated through kinematic variance analysis, where our proposed algorithm showed promising results for identifying pedaling and rest periods, which could increase the usability of classification tasks. Additionally, negative linear correlations were found between pedaling speed and decoder performance, thereby indicating that kinematic parameters between slower speeds may be easier to estimate. The results allow concluding that the use of deep learning (DL)-based methods is feasible for the estimation of lower-limb kinematic parameters during pedaling tasks using EEG signals. This study opens new possibilities for implementing controllers most robust for MMEBs and BCIs based on continuous decoding, which may allow for maximizing the degrees of freedom and personalized rehabilitation.

Convolutional neural networks can detect orthostatic hypotension in Parkinson's disease using resting-state functional near-infrared spectroscopy data

Neurological disorders such as Parkinson's disease (PD) often adversely affect the vascular system, leading to alterations in blood flow patterns. Functional near-infrared spectroscopy (fNIRS) is used to monitor hemodynamic changes via signal measurement. This study investigated the potential of using resting-state fNIRS data through a convolutional neural network (CNN) to evaluate PD with orthostatic hypotension. The CNN demonstrated significant efficacy in analyzing fNIRS data, and it outperformed the other machine learning methods. The results indicate that judicious input data selection can enhance accuracy by over 85%, while including the correlation matrix as an input further improves the accuracy to more than 90%. This study underscores the promising role of CNN-based fNIRS data analysis in the diagnosis and management of the PD. This approach enhances diagnostic accuracy, particularly in resting-state conditions, and can reduce the discomfort and risks associated with current diagnostic methods, such as the head-up tilt test.

CT-Net: an interpretable CNN-Transformer fusion network for fNIRS classification

Functional near-infrared spectroscopy (fNIRS), an optical neuroimaging technique, has been widely used in the field of brain activity recognition and brain-computer interface. Existing works have proposed deep learning-based algorithms for the fNIRS classification problem. In this paper, a novel approach based on convolutional neural network and Transformer, named CT-Net, is established to guide the deep modeling for the classification of mental arithmetic (MA) tasks. We explore the effect of data representations, and design a temporal-level combination of two raw chromophore signals to improve the data utilization and enrich the feature learning of the model. We evaluate our model on two open-access datasets and achieve the classification accuracy of 98.05% and 77.61%, respectively. Moreover, we explain our model by the gradient-weighted class activation mapping, which presents a high consistent between the contributing value of features learned by the model and the mapping of brain activity in the MA task. The results suggest the feasibility and interpretability of CT-Net for decoding MA tasks.

Novel Feature Generation for Classification of Motor Activity from Functional Near-Infrared Spectroscopy Signals Using Machine Learning

Recent research in the field of cognitive motor action decoding focuses on data acquired from Functional Near-Infrared Spectroscopy (fNIRS) and its analysis. This research aims to classify two different motor activities, namely, mental drawing (MD) and spatial navigation (SN), using fNIRS data from non-motor baseline data and other motor activities. Accurate activity detection in non-stationary signals like fNIRS is challenging and requires complex feature descriptors. As a novel framework, a new feature generation by fusion of wavelet feature, Hilbert, symlet, and Hjorth parameters is proposed for improving the accuracy of the classification. This new fused feature has statistical descriptor elements, time-localization in the frequency domain, edge feature, texture features, and phase information to detect and locate the activity accurately. Three types of independent component analysis, including FastICA, Picard, and Infomax were implemented for preprocessing which removes noises and motion artifacts. Two independent binary classifiers are designed to handle the complexity of classification in which one is responsible for mental drawing (MD) detection and the other one is spatial navigation (SN). Four different types of algorithms including nearest neighbors (KNN), Linear Discriminant Analysis (LDA), light gradient-boosting machine (LGBM), and Extreme Gradient Boosting (XGBOOST) were implemented. It has been identified that the LGBM classifier gives high accuracies-98% for mental drawing and 97% for spatial navigation. Comparison with existing research proves that the proposed method gives the highest classification accuracies. Statistical validation of the proposed new feature generation by the Kruskal-Wallis H-test and Mann-Whitney U non-parametric test proves the reliability of the proposed mechanism.

Working memory load recognition with deep learning time series classification

Working memory load (WML) is one of the widely applied signals in the areas of human-machine interaction. The precise evaluation of the WML is crucial for this kind of application. This study aims to propose a deep learning (DL) time series classification (TSC) model for inter-subject WML decoding. We used fNIRS to record the hemodynamic signals of 27 participants during visual working memory tasks. Traditional machine learning and deep time series classification algorithms were respectively used for intra-subject and inter-subject WML decoding from the collected blood oxygen signals. The intra-subject classification accuracy of LDA and SVM were 94.6% and 79.1%. Our proposed TAResnet-BiLSTM model had the highest inter-subject WML decoding accuracy, reaching 92.4%. This study provides a new idea and method for the brain-computer interface application of fNIRS in real-time WML detection.

Investigation of the neural correlation with task performance and its effect on cognitive load level classification

Electroencephalogram (EEG)-based cognitive load assessment is now an important assignment in psychological research. This type of research work is conducted by providing some mental task to the participants and their responses are counted through their EEG signal. In general assumption, it is considered that during different tasks, the cognitive workload is increased. This paper has investigated this specific idea and showed that the conventional hypothesis is not correct always. This paper showed that cognitive load can be varied according to the performance of the participants. In this paper, EEG data of 36 participants are taken against their resting and task (mental arithmetic) conditions. The features of the signal were extracted using the empirical mode decomposition (EMD) method and classified using the support vector machine (SVM) model. Based on the classification accuracy, some hypotheses are built upon the impact of subjects' performance on cognitive load. Based on some statistical consideration and graphical justification, it has been shown how the hypotheses are valid. This result will help to construct the machine learning-based model in predicting the cognitive load assessment more appropriately in a subject-independent approach.

Implementation of artificial intelligence and machine learning-based methods in brain-computer interaction

Brain-computer interfaces are used for direct two-way communication between the human brain and the computer. Brain signals contain valuable information about the mental state and brain activity of the examined subject. However, due to their non-stationarity and susceptibility to various types of interference, their processing, analysis and interpretation are challenging. For these reasons, the research in the field of brain-computer interfaces is focused on the implementation of artificial intelligence, especially in five main areas: calibration, noise suppression, communication, mental condition estimation, and motor imagery. The use of algorithms based on artificial intelligence and machine learning has proven to be very promising in these application domains, especially due to their ability to predict and learn from previous experience. Therefore, their implementation within medical technologies can contribute to more accurate information about the mental state of subjects, alleviate the consequences of serious diseases or improve the quality of life of disabled patients.

Decoding Mental Effort in a Quasi-Realistic Scenario: A Feasibility Study on Multimodal Data Fusion and Classification

Humans' performance varies due to the mental resources that are available to successfully pursue a task. To monitor users' current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications.

CGAN-rIRN: a data-augmented deep learning approach to accurate classification of mental tasks for a fNIRS-based brain-computer interface

Functional near-infrared spectroscopy (fNIRS) is increasingly used to investigate different mental tasks for brain-computer interface (BCI) control due to its excellent environmental and motion robustness. Feature extraction and classification strategy for fNIRS signal are essential to enhance the classification accuracy of voluntarily controlled BCI systems. The limitation of traditional machine learning classifiers (MLCs) lies in manual feature engineering, which is considered as one of the drawbacks that reduce accuracy. Since the fNIRS signal is a typical multivariate time series with multi-dimensionality and complexity, it makes the deep learning classifier (DLC) ideal for classifying neural activation patterns. However, the inherent bottleneck of DLCs is the requirement of substantial-scale, high-quality labeled training data and expensive computational resources to train deep networks. The existing DLCs for classifying mental tasks do not fully consider the temporal and spatial properties of fNIRS signals. Therefore, a specifically-designed DLC is desired to classify multi-tasks with high accuracy in fNIRS-BCI. To this end, we herein propose a novel data-augmented DLC to accurately classify mental tasks, which employs a convolution-based conditional generative adversarial network (CGAN) for data augmentation and a revised Inception-ResNet (rIRN) based DLC. The CGAN is utilized to generate class-specific synthetic fNIRS signals to augment the training dataset. The network architecture of rIRN is elaborately designed in accordance with the characteristics of the fNIRS signal, with serial multiple spatial and temporal feature extraction modules (FEMs), where each FEM performs deep and multi-scale feature extraction and fusion. The results of the paradigm experiments show that the proposed CGAN-rIRN approach improves the single-trial accuracy for mental arithmetic and mental singing tasks in both the data augmentation and classifier, as compared to the traditional MLCs and the commonly used DLCs. The proposed fully data-driven hybrid deep learning approach paves a promising way to improve the classification performance of volitional control fNIRS-BCI.

Lower limb exoskeleton robot and its cooperative control: A review, trends, and challenges for future research

Effective control of an exoskeleton robot (ER) using a human-robot interface is crucial for assessing the robot's movements and the force they produce to generate efficient control signals. Interestingly, certain surveys were done to show off cutting-edge exoskeleton robots. The review papers that were previously published have not thoroughly examined the control strategy, which is a crucial component of automating exoskeleton systems. As a result, this review focuses on examining the most recent developments and problems associated with exoskeleton control systems, particularly during the last few years (2017-2022). In addition, the trends and challenges of cooperative control, particularly multi-information fusion, are discussed.

Impacts of complex electromagnetic radiation and low-frequency noise exposure conditions on the cognitive function of operators

Background

Both electromagnetic radiation (EMR) and low-frequency noise (LFN) are widespread and influential environmental factors, and operators are inevitably exposed to both EMR and LFN within a complex exposure environment. The potential adverse effects of such exposure on human health must be considered seriously. This study aimed to investigate the effects of EMR and LFN on cognitive function as well as their interaction effect, which remain unclear.

Methods

Sixty young male college students were randomly grouped and experiments were conducted with a 2 × 2 factorial design in a shielded chamber. Mental workload (MWL) levels of the study subjects were measured and assessed using the NASA-task load index (TLX) subjective scale, an n-back task paradigm, and the functional near-infrared spectroscopy (fNIRS) imaging technique.

Results

For the 3-back task, the NASA-TLX subjective scale revealed a statistically significant main effect of LFN intensity, which enhanced the subjects' MWL level (F = 8.716, p < 0.01). Behavioral performance revealed that EMR intensity (430.1357 MHz, 10.75 W/m²) and LFN intensity (0-200 Hz, 72.9 dB) had a synergistic interaction effect, and the correct response time was statistically significantly prolonged by the combined exposure (F = 4.343, p < 0.05). The fNIRS imaging technique revealed a synergistic interaction effect between operational EMR intensity and operational LFN intensity, with statistically significant effects on the activation levels in the left and right dorsolateral prefrontal cortex (DLPFC). The mean β values of DLPFC were significantly increased (L-DLPFC F = 5.391, p < 0.05, R-DLPFC F = 4.222, p < 0.05), and the relative concentrations of oxyhemoglobin in the DLPFC were also significantly increased (L-DLPFC F = 4.925, p < 0.05, R-DLPFC F = 9.715, p < 0.01).

Conclusion

We found a statistically significant interaction effect between EMR (430.1357 MHz, 10.75 W/m²) and LFN (0-200 Hz, 72.9 dB) when simultaneously exposing subjects to both for 30 min. We conclude that exposure to this complex environment can cause a statistically significant increase in the MWL level of operators, and even alterations in their cognitive function.

Image Classification Based on Light Convolutional Neural Network Using Pulse Couple Neural Network

Recently, most image classification studies solicit the intervention of convolutional neural networks because these DL-based classification methods generally outperform other methodologies with higher accuracy. However, this type of deep learning networks require many parameters and have a complex structure with multiple convolutional and pooling layers depending on the objective. These layers compute a large volume of data and it may impact the processing time and the performance. Therefore, this paper proposes a new method of image classification based on the light convolutional neural network. It consists of replacing the feature extraction layers of standard convolutional neural network with a single pulse coupled neural network by introducing the notion of foveation. This module provides the feature map of input image and the data compression using Discrete Wavelet Transform which is an optional step depending on the information quantity of this signature. The fully connected neural network, which has six hidden layers, classifies the image. With this technique, the computation time is reduced, and the network architecture is identical and simple independent of the type of dataset. The number of parameter is less than that in current research. The proposed method was validated with different dataset such as Caltech-101, Caltech-256, CIFAR-10, CIFAR-100, and ImageNet, and the accuracy reaches 92%, 90%, 99%, 94%, and 91%, respectively, which are better than the previous related works.

Combining robust level extraction and unsupervised adaptive classification for high-accuracy fNIRS-BCI: An evidence on single-trial differentiation between mentally arithmetic- and singing-tasks

Functional near-infrared spectroscopy (fNIRS) is a safe and non-invasive optical imaging technique that is being increasingly used in brain-computer interfaces (BCIs) to recognize mental tasks. Unlike electroencephalography (EEG) which directly measures neural activation, fNIRS signals reflect neurovascular-coupling inducing hemodynamic response that can be slow in time and varying in the pattern. The established classifiers extend the EEG-ones by mostly employing the feature based supervised models such as the support vector machine (SVM) and linear discriminant analysis (LDA), and fail to timely characterize the level-sensitive hemodynamic pattern. A dedicated classifier is desired for intentional activity recognition of fNIRS-BCI, including the adaptive acquisition of response relevant features and accurate discrimination of implied ideas. To this end, we herein propose a specifically-designed joint adaptive classification method that combines a Kalman filtering (KF) for robust level extraction and an adaptive Gaussian mixture model (a-GMM) for enhanced pattern recognition. The simulative investigations and paradigm experiments have shown that the proposed KF/a-GMM classification method can effectively track the random variations of task-evoked brain activation patterns, and improve the accuracy of single-trial classification task of mental arithmetic vs. mental singing, as compared to the conventional methods, e.g., those that employ combinations of the band-pass filtering (BPF) based feature extractors (mean, slope, and variance, etc.) and the classical recognizers (GMM, SVM, and LDA). The proposed approach paves a promising way for developing the real-time fNIRS-BCI technique.

Deep learning in fNIRS: a review

Significance: Optical neuroimaging has become a well-established clinical and research tool to monitor cortical activations in the human brain. It is notable that outcomes of functional near-infrared spectroscopy (fNIRS) studies depend heavily on the data processing pipeline and classification model employed. Recently, deep learning (DL) methodologies have demonstrated fast and accurate performances in data processing and classification tasks across many biomedical fields. Aim: We aim to review the emerging DL applications in fNIRS studies. Approach: We first introduce some of the commonly used DL techniques. Then, the review summarizes current DL work in some of the most active areas of this field, including brain-computer interface, neuro-impairment diagnosis, and neuroscience discovery. Results: Of the 63 papers considered in this review, 32 report a comparative study of DL techniques to traditional machine learning techniques where 26 have been shown outperforming the latter in terms of the classification accuracy. In addition, eight studies also utilize DL to reduce the amount of preprocessing typically done with fNIRS data or increase the amount of data via data augmentation. Conclusions: The application of DL techniques to fNIRS studies has shown to mitigate many of the hurdles present in fNIRS studies such as lengthy data preprocessing or small sample sizes while achieving comparable or improved classification accuracy.

Brain augmentation and neuroscience technologies: current applications, challenges, ethics and future prospects

Ever since the dawn of antiquity, people have strived to improve their cognitive abilities. From the advent of the wheel to the development of artificial intelligence, technology has had a profound leverage on civilization. Cognitive enhancement or augmentation of brain functions has become a trending topic both in academic and public debates in improving physical and mental abilities. The last years have seen a plethora of suggestions for boosting cognitive functions and biochemical, physical, and behavioral strategies are being explored in the field of cognitive enhancement. Despite expansion of behavioral and biochemical approaches, various physical strategies are known to boost mental abilities in diseased and healthy individuals. Clinical applications of neuroscience technologies offer alternatives to pharmaceutical approaches and devices for diseases that have been fatal, so far. Importantly, the distinctive aspect of these technologies, which shapes their existing and anticipated participation in brain augmentations, is used to compare and contrast them. As a preview of the next two decades of progress in brain augmentation, this article presents a plausible estimation of the many neuroscience technologies, their virtues, demerits, and applications. The review also focuses on the ethical implications and challenges linked to modern neuroscientific technology. There are times when it looks as if ethics discussions are more concerned with the hypothetical than with the factual. We conclude by providing recommendations for potential future studies and development areas, taking into account future advancements in neuroscience innovation for brain enhancement, analyzing historical patterns, considering neuroethics and looking at other related forecasts.

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.

A General and Scalable Vision Framework for Functional Near-Infrared Spectroscopy Classification

Functional near-infrared spectroscopy (fNIRS), a non-invasive optical technique, is widely used to monitor brain activities for disease diagnosis and brain-computer interfaces (BCIs). Deep learning-based fNIRS classification faces three major barriers: limited datasets, confusing evaluation criteria, and domain barriers. We apply more appropriate evaluation methods to three open-access datasets to solve the first two barriers. For domain barriers, we propose a general and scalable vision fNIRS framework that converts multi-channel fNIRS signals into multi-channel virtual images using the Gramian angular difference field (GADF). We use the framework to train state-of-the-art visual models from computer vision (CV) within a few minutes, and the classification performance is competitive with the latest fNIRS models. In cross-validation experiments, visual models achieve the highest average classification results of 78.68% and 73.92% on mental arithmetic and word generation tasks, respectively. Although visual models are slightly lower than the fNIRS models on unilateral finger- and foot-tapping tasks, the F1-score and kappa coefficient indicate that these differences are insignificant in subject-independent experiments. Furthermore, we study fNIRS signal representations and the classification performance of sequence-to-image methods. We hope to introduce rich achievements from the CV domain to improve fNIRS classification research.

An Evaluation of the EEG Alpha-to-Theta and Theta-to-Alpha Band Ratios as Indexes of Mental Workload

Many research works indicate that EEG bands, specifically the alpha and theta bands, have been potentially helpful cognitive load indicators. However, minimal research exists to validate this claim. This study aims to assess and analyze the impact of the alpha-to-theta and the theta-to-alpha band ratios on supporting the creation of models capable of discriminating self-reported perceptions of mental workload. A dataset of raw EEG data was utilized in which 48 subjects performed a resting activity and an induced task demanding exercise in the form of a multitasking SIMKAP test. Band ratios were devised from frontal and parietal electrode clusters. Building and model testing was done with high-level independent features from the frequency and temporal domains extracted from the computed ratios over time. Target features for model training were extracted from the subjective ratings collected after resting and task demand activities. Models were built by employing Logistic Regression, Support Vector Machines and Decision Trees and were evaluated with performance measures including accuracy, recall, precision and f1-score. The results indicate high classification accuracy of those models trained with the high-level features extracted from the alpha-to-theta ratios and theta-to-alpha ratios. Preliminary results also show that models trained with logistic regression and support vector machines can accurately classify self-reported perceptions of mental workload. This research contributes to the body of knowledge by demonstrating the richness of the information in the temporal, spectral and statistical domains extracted from the alpha-to-theta and theta-to-alpha EEG band ratios for the discrimination of self-reported perceptions of mental workload.

Training Monitoring in Sports: It Is Time to Embrace Cognitive Demand

Appropriate training burden monitoring is still a challenge for the support staff, athletes, and coaches. Extensive research has been done in recent years that proposes several external and internal indicators. Among all measurements, the importance of cognitive factors has been indicated but has never been really considered in the training monitoring process. While there is strong evidence supporting the use of cognitive demand indicators in cognitive neuroscience, their importance in training monitoring for multiple sports settings must be better emphasized. The aims of this scoping review are to (1) provide an overview of the cognitive demand concept beside the physical demand in training; (2) highlight the current methods for assessing cognitive demand in an applied setting to sports in part through a neuroergonomics approach; (3) show how cognitive demand metrics can be exploited and applied to our better understanding of fatigue, sport injury, overtraining and individual performance capabilities. This review highlights also the potential new ways of brain imaging approaches for monitoring in situ. While assessment of cognitive demand is still in its infancy in sport, it may represent a very fruitful approach if applied with rigorous protocols and deep knowledge of both the neurobehavioral and cognitive aspects. It is time now to consider the cognitive demand to avoid underestimating the total training burden and its management.

Analyzing Classification Performance of fNIRS-BCI for Gait Rehabilitation Using Deep Neural Networks

This research presents a brain-computer interface (BCI) framework for brain signal classification using deep learning (DL) and machine learning (ML) approaches on functional near-infrared spectroscopy (fNIRS) signals. fNIRS signals of motor execution for walking and rest tasks are acquired from the primary motor cortex in the brain's left hemisphere for nine subjects. DL algorithms, including convolutional neural networks (CNNs), long short-term memory (LSTM), and bidirectional LSTM (Bi-LSTM) are used to achieve average classification accuracies of 88.50%, 84.24%, and 85.13%, respectively. For comparison purposes, three conventional ML algorithms, support vector machine (SVM), k-nearest neighbor (k-NN), and linear discriminant analysis (LDA) are also used for classification, resulting in average classification accuracies of 73.91%, 74.24%, and 65.85%, respectively. This study successfully demonstrates that the enhanced performance of fNIRS-BCI can be achieved in terms of classification accuracy using DL approaches compared to conventional ML approaches. Furthermore, the control commands generated by these classifiers can be used to initiate and stop the gait cycle of the lower limb exoskeleton for gait rehabilitation.

Novel fNIRS study on homogeneous symmetric feature-based transfer learning for brain-computer interface

The brain-computer interface (BCI) provides an alternate means of communication between the brain and external devices by recognizing the brain activities and translating them into external commands. The functional Near-Infrared Spectroscopy (fNIRS) is becoming popular as a non-invasive modality for brain activity detection. The recent trends show that deep learning has significantly enhanced the performance of the BCI systems. But the inherent bottleneck for deep learning (in the domain of BCI) is the requirement of the vast amount of training data, lengthy recalibrating time, and expensive computational resources for training deep networks. Building a high-quality, large-scale annotated dataset for deep learning-based BCI systems is exceptionally tedious, complex, and expensive. This study investigates the novel application of transfer learning for fNIRS-based BCI to solve three objective functions (concerns), i.e., the problem of insufficient training data, reduced training time, and increased accuracy. We applied symmetric homogeneous feature-based transfer learning on convolutional neural network (CNN) designed explicitly for fNIRS data collected from twenty-six (26) participants performing the n-back task. The results suggested that the proposed method achieves the maximum saturated accuracy sooner and outperformed the traditional CNN model on averaged accuracy by 25.58% in the exact duration of training time, reducing the training time, recalibrating time, and computational resources.

Optimized electroencephalogram and functional near-infrared spectroscopy-based mental workload detection method for practical applications

Background

Mental workload is a critical consideration in complex man–machine systems design. Among various mental workload detection techniques, multimodal detection techniques integrating electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals have attracted considerable attention. However, existing EEG–fNIRS-based mental workload detection methods have certain defects, such as complex signal acquisition channels and low detection accuracy, which restrict their practical application.

Methods

The signal acquisition configuration was optimized by analyzing the feature importance in mental workload recognition model and a more accurate and convenient EEG–fNIRS-based mental workload detection method was constructed. A classical Multi-Task Attribute Battery (MATB) task was conducted with 20 participating volunteers. Subjective scale data, 64-channel EEG data, and two-channel fNIRS data were collected.

Results

A higher number of EEG channels correspond to higher detection accuracy. However, there is no obvious improvement in accuracy once the number of EEG channels reaches 26, with a four-level mental workload detection accuracy of 76.25 ± 5.21%. Partial results of physiological analysis verify the results of previous studies, such as that the θ power of EEG and concentration of O₂Hb in the prefrontal region increase while the concentration of HHb decreases with task difficulty. It was further observed, for the first time, that the energy of each band of EEG signals was significantly different in the occipital lobe region, and the power of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta_{1}$$\end{document}β1 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta_{2}$$\end{document}β2 bands in the occipital region increased significantly with task difficulty. The changing range and the mean amplitude of O₂Hb in high-difficulty tasks were significantly higher compared with those in low-difficulty tasks.

Conclusions

The channel configuration of EEG–fNIRS-based mental workload detection was optimized to 26 EEG channels and two frontal fNIRS channels. A four-level mental workload detection accuracy of 76.25 ± 5.21% was obtained, which is higher than previously reported results. The proposed configuration can promote the application of mental workload detection technology in military, driving, and other complex human–computer interaction systems.

Transformer Model for Functional Near-Infrared Spectroscopy Classification

Functional near-infrared spectroscopy (fNIRS) is a promising neuroimaging technology. The fNIRS classification problem has always been the focus of the brain-computer interface (BCI). Inspired by the success of Transformer based on self-attention mechanism in the fields of natural language processing and computer vision, we propose an fNIRS classification network based on Transformer, named fNIRS-T. We explore the spatial-level and channel-level representation of fNIRS signals to improve data utilization and network representation capacity. Besides, a preprocessing module, which consists of one-dimensional average pooling and layer normalization, is designed to replace filtering and baseline correction of data preprocessing. It makes fNIRS-T an end-to-end network, called fNIRS-PreT. Compared with traditional machine learning classifiers, convolutional neural network (CNN), and long short-term memory (LSTM), the proposed models obtain the best accuracy on three open-access datasets. Specifically, in the most extensive ternary classification task (30 subjects) that includes three types of overt movements, fNIRS-T, CNN, and LSTM obtain 75.49%, 72.89%, and 61.94% on test sets, respectively. Compared to traditional classifiers, fNIRS-T is at least 27.41% higher than statistical features and 6.79% higher than well-designed features. In the individual subject experiment of the ternary classification task, fNIRS-T achieves an average subject accuracy of 78.22% and surpasses CNN and LSTM by a large margin of +4.75% and +11.33%. fNIRS-PreT using raw data also achieves competitive performance to fNIRS-T. Therefore, the proposed models improve the performance of fNIRS-based BCI significantly.

Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain-Computer Interface

A passive brain-computer interface (BCI) based upon functional near-infrared spectroscopy (fNIRS) brain signals is used for earlier detection of human drowsiness during driving tasks. This BCI modality acquired hemodynamic signals of 13 healthy subjects from the right dorsolateral prefrontal cortex (DPFC) of the brain. Drowsiness activity is recorded using a continuous-wave fNIRS system and eight channels over the right DPFC. During the experiment, sleep-deprived subjects drove a vehicle in a driving simulator while their cerebral oxygen regulation (CORE) state was continuously measured. Vector phase analysis (VPA) was used as a classifier to detect drowsiness state along with sleep stage-based threshold criteria. Extensive training and testing with various feature sets and classifiers are done to justify the adaptation of threshold criteria for any subject without requiring recalibration. Three statistical features (mean oxyhemoglobin, signal peak, and the sum of peaks) along with six VPA features (trajectory slopes of VPA indices) were used. The average accuracies for the five classifiers are 90.9% for discriminant analysis, 92.5% for support vector machines, 92.3% for nearest neighbors, 92.4% for both decision trees, and ensembles over all subjects' data. Trajectory slopes of CORE vector magnitude and angle: m(|R|) and m(∠R) are the best-performing features, along with ensemble classifier with the highest accuracy of 95.3% and minimum computation time of 40 ms. The statistical significance of the results is validated with a p-value of less than 0.05. The proposed passive BCI scheme demonstrates a promising technique for online drowsiness detection using VPA along with sleep stage classification.

Motor Training Using Mental Workload (MWL) With an Assistive Soft Exoskeleton System: A Functional Near-Infrared Spectroscopy (fNIRS) Study for Brain-Machine Interface (BMI)

Mental workload is a neuroergonomic human factor, which is widely used in planning a system's safety and areas like brain-machine interface (BMI), neurofeedback, and assistive technologies. Robotic prosthetics methodologies are employed for assisting hemiplegic patients in performing routine activities. Assistive technologies' design and operation are required to have an easy interface with the brain with fewer protocols, in an attempt to optimize mobility and autonomy. The possible answer to these design questions may lie in neuroergonomics coupled with BMI systems. In this study, two human factors are addressed: designing a lightweight wearable robotic exoskeleton hand that is used to assist the potential stroke patients with an integrated portable brain interface using mental workload (MWL) signals acquired with portable functional near-infrared spectroscopy (fNIRS) system. The system may generate command signals for operating a wearable robotic exoskeleton hand using two-state MWL signals. The fNIRS system is used to record optical signals in the form of change in concentration of oxy and deoxygenated hemoglobin (HbO and HbR) from the pre-frontal cortex (PFC) region of the brain. Fifteen participants participated in this study and were given hand-grasping tasks. Two-state MWL signals acquired from the PFC region of the participant's brain are segregated using machine learning classifier-support vector machines (SVM) to utilize in operating a robotic exoskeleton hand. The maximum classification accuracy is 91.31%, using a combination of mean-slope features with an average information transfer rate (ITR) of 1.43. These results show the feasibility of a two-state MWL (fNIRS-based) robotic exoskeleton hand (BMI system) for hemiplegic patients assisting in the physical grasping tasks.

Subject-Independent Functional Near-Infrared Spectroscopy-Based Brain-Computer Interfaces Based on Convolutional Neural Networks

Functional near-infrared spectroscopy (fNIRS) has attracted increasing attention in the field of brain–computer interfaces (BCIs) owing to their advantages such as non-invasiveness, user safety, affordability, and portability. However, fNIRS signals are highly subject-specific and have low test-retest reliability. Therefore, individual calibration sessions need to be employed before each use of fNIRS-based BCI to achieve a sufficiently high performance for practical BCI applications. In this study, we propose a novel deep convolutional neural network (CNN)-based approach for implementing a subject-independent fNIRS-based BCI. A total of 18 participants performed the fNIRS-based BCI experiments, where the main goal of the experiments was to distinguish a mental arithmetic task from an idle state task. Leave-one-subject-out cross-validation was employed to evaluate the average classification accuracy of the proposed subject-independent fNIRS-based BCI. As a result, the average classification accuracy of the proposed method was reported to be 71.20 ± 8.74%, which was higher than the threshold accuracy for effective BCI communication (70%) as well as that obtained using conventional shrinkage linear discriminant analysis (65.74 ± 7.68%). To achieve a classification accuracy comparable to that of the proposed subject-independent fNIRS-based BCI, 24 training trials (of approximately 12 min) were necessary for the traditional subject-dependent fNIRS-based BCI. It is expected that our CNN-based approach would reduce the necessity of long-term individual calibration sessions, thereby enhancing the practicality of fNIRS-based BCIs significantly.

Hybrid EEG-fNIRS BCI Fusion Using Multi-Resolution Singular Value Decomposition (MSVD)

Brain-computer interface (BCI) multi-modal fusion has the potential to generate multiple commands in a highly reliable manner by alleviating the drawbacks associated with single modality. In the present work, a hybrid EEG-fNIRS BCI system—achieved through a fusion of concurrently recorded electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals—is used to overcome the limitations of uni-modality and to achieve higher tasks classification. Although the hybrid approach enhances the performance of the system, the improvements are still modest due to the lack of availability of computational approaches to fuse the two modalities. To overcome this, a novel approach is proposed using Multi-resolution singular value decomposition (MSVD) to achieve system- and feature-based fusion. The two approaches based up different features set are compared using the KNN and Tree classifiers. The results obtained through multiple datasets show that the proposed approach can effectively fuse both modalities with improvement in the classification accuracy.

A Bipolar-Channel Hybrid Brain-Computer Interface System for Home Automation Control Utilizing Steady-State Visually Evoked Potential and Eye-Blink Signals

The goal of this study was to develop and validate a hybrid brain-computer interface (BCI) system for home automation control. Over the past decade, BCIs represent a promising possibility in the field of medical (e.g., neuronal rehabilitation), educational, mind reading, and remote communication. However, BCI is still difficult to use in daily life because of the challenges of the unfriendly head device, lower classification accuracy, high cost, and complex operation. In this study, we propose a hybrid BCI system for home automation control with two brain signals acquiring electrodes and simple tasks, which only requires the subject to focus on the stimulus and eye blink. The stimulus is utilized to select commands by generating steady-state visually evoked potential (SSVEP). The single eye blinks (i.e., confirm the selection) and double eye blinks (i.e., deny and re-selection) are employed to calibrate the SSVEP command. Besides that, the short-time Fourier transform and convolution neural network algorithms are utilized for feature extraction and classification, respectively. The results show that the proposed system could provide 38 control commands with a 2 s time window and a good accuracy (i.e., 96.92%) using one bipolar electroencephalogram (EEG) channel. This work presents a novel BCI approach for the home automation application based on SSVEP and eye blink signals, which could be useful for the disabled. In addition, the provided strategy of this study—a friendly channel configuration (i.e., one bipolar EEG channel), high accuracy, multiple commands, and short response time—might also offer a reference for the other BCI controlled applications.