Classification of prefrontal and motor cortex signals for three-class fNIRS–BCI

Brain-computer interface paradigms and neural coding

Brain signal patterns generated in the central nervous system of brain-computer interface (BCI) users are closely related to BCI paradigms and neural coding. In BCI systems, BCI paradigms and neural coding are critical elements for BCI research. However, so far there have been few references that clearly and systematically elaborated on the definition and design principles of the BCI paradigm as well as the definition and modeling principles of BCI neural coding. Therefore, these contents are expounded and the existing main BCI paradigms and neural coding are introduced in the review. Finally, the challenges and future research directions of BCI paradigm and neural coding were discussed, including user-centered design and evaluation for BCI paradigms and neural coding, revolutionizing the traditional BCI paradigms, breaking through the existing techniques for collecting brain signals and combining BCI technology with advanced AI technology to improve brain signal decoding performance. It is expected that the review will inspire innovative research and development of the BCI paradigm and neural coding.

Research on Emotion Recognition Method of Cerebral Blood Oxygen Signal Based on CNN-Transformer Network

In recent years, research on emotion recognition has become more and more popular, but there are few studies on emotion recognition based on cerebral blood oxygen signals. Since the electroencephalogram (EEG) is easily disturbed by eye movement and the portability is not high, this study uses a more comfortable and convenient functional near-infrared spectroscopy (fNIRS) system to record brain signals from participants while watching three different types of video clips. During the experiment, the changes in cerebral blood oxygen concentration in the 8 channels of the prefrontal cortex of the brain were collected and analyzed. We processed and divided the collected cerebral blood oxygen data, and used multiple classifiers to realize the identification of the three emotional states of joy, neutrality, and sadness. Since the classification accuracy of the convolutional neural network (CNN) in this research is not significantly superior to that of the XGBoost algorithm, this paper proposes a CNN-Transformer network based on the characteristics of time series data to improve the classification accuracy of ternary emotions. The network first uses convolution operations to extract channel features from multi-channel time series, then the features and the output information of the fully connected layer are input to the Transformer netork structure, and its multi-head attention mechanism is used to focus on different channel domain information, which has better spatiality. The experimental results show that the CNN-Transformer network can achieve 86.7% classification accuracy for ternary emotions, which is about 5% higher than the accuracy of CNN, and this provides some help for other research in the field of emotion recognition based on time series data such as fNIRS.

It takes two (seconds): decreasing encoding time for two-choice functional near-infrared spectroscopy brain-computer interface communication

Significance

Brain-computer interfaces (BCIs) can provide severely motor-impaired patients with a motor-independent communication channel. Functional near-infrared spectroscopy (fNIRS) constitutes a promising BCI-input modality given its high mobility, safety, user comfort, cost-efficiency, and relatively low motion sensitivity.

Aim

The present study aimed at developing an efficient and convenient two-choice fNIRS communication BCI by implementing a relatively short encoding time (2 s), considerably increasing communication speed, and decreasing the cognitive load of BCI users.

Approach

To encode binary answers to 10 biographical questions, 10 healthy adults repeatedly performed a combined motor-speech imagery task within 2 different time windows guided by auditory instructions. Each answer-encoding run consisted of 10 trials. Answers were decoded during the ongoing experiment from the time course of the individually identified most-informative fNIRS channel-by-chromophore combination.

Results

The answers of participants were decoded online with an accuracy of 85.8% (run-based group mean). Post-hoc analysis yielded an average single-trial accuracy of 68.1%. Analysis of the effect of number of trial repetitions showed that the best information-transfer rate could be obtained by combining four encoding trials.

Conclusions

The study demonstrates that an encoding time as short as 2 s can enable immediate, efficient, and convenient fNIRS-BCI communication.

The Effects of Different Brain Regions on fNIRS-based Task-state Detection in Speech Imagery

Brain-computer interface (BCI) based on speech imagery can decode users' verbal intent and help people with motor disabilities communicate naturally. Functional near-infrared spectroscopy (fNIRS) is a commonly used brain signal acquisition method. Asynchronous BCI can response to control commands at any time, which provides great convenience for users. Task state detection, defined as identifying whether user starts or continues covertly articulating, plays an important role in speech imagery BCIs. To better distinguish task state from idle state during speech imagery, this work used fNIRS signals from different brain regions to study the effects of different brain regions on task state detection accuracy. The imagined tonal syllables included four lexical tones and four vowels in Mandarin Chinese. The brain regions that were measured included Broca's area, Wernicke's area, Superior temporal cortex and Motor cortex. Task state detection accuracies of imagining tonal monosyllables with four different tones were analyzed. The average accuracy of four speech imagery tasks based on the whole brain was 0.67 and it was close to 0.69, which was the average accuracy based on Broca's area. The accuracies of Broca's area and the whole brain were significantly higher than those of other brain regions. The findings of this work demonstrated that using a few channels of Broca's area could result in a similar task state detection accuracy to that using all the channels of the brain. Moreover, it was discovered that speech imagery with tone 2/3 tasks yielded higher task state detection accuracy than speech imagery with other tones.

Benchmarking framework for machine learning classification from fNIRS data

Background

While efforts to establish best practices with functional near infrared spectroscopy (fNIRS) signal processing have been published, there are still no community standards for applying machine learning to fNIRS data. Moreover, the lack of open source benchmarks and standard expectations for reporting means that published works often claim high generalisation capabilities, but with poor practices or missing details in the paper. These issues make it hard to evaluate the performance of models when it comes to choosing them for brain-computer interfaces.

Methods

We present an open-source benchmarking framework, BenchNIRS, to establish a best practice machine learning methodology to evaluate models applied to fNIRS data, using five open access datasets for brain-computer interface (BCI) applications. The BenchNIRS framework, using a robust methodology with nested cross-validation, enables researchers to optimise models and evaluate them without bias. The framework also enables us to produce useful metrics and figures to detail the performance of new models for comparison. To demonstrate the utility of the framework, we present a benchmarking of six baseline models [linear discriminant analysis (LDA), support-vector machine (SVM), k-nearest neighbours (kNN), artificial neural network (ANN), convolutional neural network (CNN), and long short-term memory (LSTM)] on the five datasets and investigate the influence of different factors on the classification performance, including: number of training examples and size of the time window of each fNIRS sample used for classification. We also present results with a sliding window as opposed to simple classification of epochs, and with a personalised approach (within subject data classification) as opposed to a generalised approach (unseen subject data classification).

Results and discussion

Results show that the performance is typically lower than the scores often reported in literature, and without great differences between models, highlighting that predicting unseen data remains a difficult task. Our benchmarking framework provides future authors, who are achieving significant high classification scores, with a tool to demonstrate the advances in a comparable way. To complement our framework, we contribute a set of recommendations for methodology decisions and writing papers, when applying machine learning to fNIRS data.

Explainable artificial intelligence model to predict brain states from fNIRS signals

Objective: Most Deep Learning (DL) methods for the classification of functional Near-Infrared Spectroscopy (fNIRS) signals do so without explaining which features contribute to the classification of a task or imagery. An explainable artificial intelligence (xAI) system that can decompose the Deep Learning mode's output onto the input variables for fNIRS signals is described here. Approach: We propose an xAI-fNIRS system that consists of a classification module and an explanation module. The classification module consists of two separately trained sliding window-based classifiers, namely, (i) 1-D Convolutional Neural Network (CNN); and (ii) Long Short-Term Memory (LSTM). The explanation module uses SHAP (SHapley Additive exPlanations) to explain the CNN model's output in terms of the model's input. Main results: We observed that the classification module was able to classify two types of datasets: (a) Motor task (MT), acquired from three subjects; and (b) Motor imagery (MI), acquired from 29 subjects, with an accuracy of over 96% for both CNN and LSTM models. The explanation module was able to identify the channels contributing the most to the classification of MI or MT and therefore identify the channel locations and whether they correspond to oxy- or deoxy-hemoglobin levels in those locations. Significance: The xAI-fNIRS system can distinguish between the brain states related to overt and covert motor imagery from fNIRS signals with high classification accuracy and is able to explain the signal features that discriminate between the brain states of interest.

Machine learning in biosignals processing for mental health: A narrative review

Machine Learning (ML) offers unique and powerful tools for mental health practitioners to improve evidence-based psychological interventions and diagnoses. Indeed, by detecting and analyzing different biosignals, it is possible to differentiate between typical and atypical functioning and to achieve a high level of personalization across all phases of mental health care. This narrative review is aimed at presenting a comprehensive overview of how ML algorithms can be used to infer the psychological states from biosignals. After that, key examples of how they can be used in mental health clinical activity and research are illustrated. A description of the biosignals typically used to infer cognitive and emotional correlates (e.g., EEG and ECG), will be provided, alongside their application in Diagnostic Precision Medicine, Affective Computing, and brain-computer Interfaces. The contents will then focus on challenges and research questions related to ML applied to mental health and biosignals analysis, pointing out the advantages and possible drawbacks connected to the widespread application of AI in the medical/mental health fields. The integration of mental health research and ML data science will facilitate the transition to personalized and effective medicine, and, to do so, it is important that researchers from psychological/ medical disciplines/health care professionals and data scientists all share a common background and vision of the current research.

Early-stage fusion of EEG and fNIRS improves classification of motor imagery

Introduction

Many research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which were generally customized and hard to be generalized. How to effectively utilize information from the two modalities was still unclear.

Methods

In this paper, we conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS. A Y-shaped neural network was proposed and evaluated on an open dataset, which fuses the bimodal information in different stages.

Results

The results suggests that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage and late-stage fusion network configuration (N = 57, P < 0.05). With the proposed framework, the average accuracy of 29 participants reaches 76.21% in the left-or-right hand motor imagery task in leave-one-out cross-validation, using bi-modal data as network inputs respectively, which is in the same level as the state-of-the-art hybrid BCI methods based on EEG and fNIRS data.

Functional near-infrared spectroscopy is a useful tool for multi-perspective psychobiological study of neurophysiological correlates of parenting behaviour

The quality of the relationship between caregiver and child has long-term effects on the cognitive and socio-emotional development of children. A process involved in human parenting is the bio-behavioural synchrony that occurs between the partners in the relationship during interaction. Through interaction, bio-behavioural synchronicity allows the adaptation of the physiological systems of the parent to those of the child and promotes the positive development and modelling of the child's social brain. The role of bio-behavioural synchrony in building social bonds could be investigated using functional near-infrared spectroscopy (fNIRS). In this paper we have (a) highlighted the importance of the quality of the caregiver-child relationship for the child's cognitive and socio-emotional development, as well as the relevance of infantile stimuli in the activation of parenting behaviour; (b) discussed the tools used in the study of the neurophysiological substrates of the parental response; (c) proposed fNIRS as a particularly suitable tool for the study of parental responses; and (d) underlined the need for a multi-systemic psychobiological approach to understand the mechanisms that regulate caregiver-child interactions and their bio-behavioural synchrony. We propose to adopt a multi-system psychobiological approach to the study of parental behaviour and social interaction.

An online hybrid BCI combining SSVEP and EOG-based eye movements

Hybrid brain-computer interface (hBCI) refers to a system composed of a single-modality BCI and another system. In this paper, we propose an online hybrid BCI combining steady-state visual evoked potential (SSVEP) and eye movements to improve the performance of BCI systems. Twenty buttons corresponding to 20 characters are evenly distributed in the five regions of the GUI and flash at the same time to arouse SSVEP. At the end of the flash, the buttons in the four regions move in different directions, and the subject continues to stare at the target with eyes to generate the corresponding eye movements. The CCA method and FBCCA method were used to detect SSVEP, and the electrooculography (EOG) waveform was used to detect eye movements. Based on the EOG features, this paper proposes a decision-making method based on SSVEP and EOG, which can further improve the performance of the hybrid BCI system. Ten healthy students took part in our experiment, and the average accuracy and information transfer rate of the system were 94.75% and 108.63 bits/min, respectively.

A functional near-infrared spectroscopy investigation of item-method directed forgetting

Whether directed forgetting is passive or active remains debated. Using functional near-infrared spectroscopy (fNIRS), blood-oxygen level-dependent responses of intentional forgetting were investigated in the item-method directed forgetting (DF) paradigm. During the study phase, each word was followed by a random remembering or forgetting cue indicating whether the word is to be remembered (TBR) or to be forgotten (TBF). A recognition test was used in the test phase and four cue-response conditions were obtained: remembering/forgetting cues associated with the subsequently remembered (TBR-r/TBF-r) or forgotten (TBR-f/TBF-f) words. Data from 16 healthy adult participants showed a DF effect. The fNIRS data revealed that, during the 5-9 s time window, the oxygenate hemoglobin (oxy-Hb) levels were higher during intentional forgetting compared to intentional remembering in the left inferior frontal (TBF-f vs. TBR-f) and right superior frontal gyrus (TBF-r vs. TBR-r), indicating more frontal inhibition involved during intentional forgetting. During the 9-11 s time window, the oxy-Hb level in the frontal and parietal gyrus was higher for forgetting than remembering cues, indicating that the TBF words might be automatically encoded. In sum, the TBF words might receive inhibition control triggered by forgetting cues and then be automatically encoded with the increase of the post-cue interval.

SPD-CNN: A plain CNN-based model using the symmetric positive definite matrices for cross-subject EEG classification with meta-transfer-learning

The electroencephalography (EEG) signals are easily contaminated by various artifacts and noise, which induces a domain shift in each subject and significant pattern variability among different subjects. Therefore, it hinders the improvement of EEG classification accuracy in the cross-subject learning scenario. Convolutional neural networks (CNNs) have been extensively applied to EEG-based Brain-Computer Interfaces (BCIs) by virtue of the capability of performing automatic feature extraction and classification. However, they have been mainly applied to the within-subject classification which would consume lots of time for training and calibration. Thus, it limits the further applications of CNNs in BCIs. In order to build a robust classification algorithm for a calibration-less BCI system, we propose an end-to-end model that transforms the EEG signals into symmetric positive definite (SPD) matrices and captures the features of SPD matrices by using a CNN. To avoid the time-consuming calibration and ensure the application of the proposed model, we use the meta-transfer-learning (MTL) method to learn the essential features from different subjects. We validate our model by making extensive experiments on three public motor-imagery datasets. The experimental results demonstrate the effectiveness of our proposed method in the cross-subject learning scenario.

Motion Artifacts Correction from Single-Channel EEG and fNIRS Signals Using Novel Wavelet Packet Decomposition in Combination with Canonical Correlation Analysis

The electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals, highly non-stationary in nature, greatly suffers from motion artifacts while recorded using wearable sensors. Since successful detection of various neurological and neuromuscular disorders is greatly dependent upon clean EEG and fNIRS signals, it is a matter of utmost importance to remove/reduce motion artifacts from EEG and fNIRS signals using reliable and robust methods. In this regard, this paper proposes two robust methods: (i) Wavelet packet decomposition (WPD) and (ii) WPD in combination with canonical correlation analysis (WPD-CCA), for motion artifact correction from single-channel EEG and fNIRS signals. The efficacy of these proposed techniques is tested using a benchmark dataset and the performance of the proposed methods is measured using two well-established performance matrices: (i) difference in the signal to noise ratio ( ) and (ii) percentage reduction in motion artifacts ( ). The proposed WPD-based single-stage motion artifacts correction technique produces the highest average (29.44 dB) when db2 wavelet packet is incorporated whereas the greatest average (53.48%) is obtained using db1 wavelet packet for all the available 23 EEG recordings. Our proposed two-stage motion artifacts correction technique, i.e., the WPD-CCA method utilizing db1 wavelet packet has shown the best denoising performance producing an average and values of 30.76 dB and 59.51%, respectively, for all the EEG recordings. On the other hand, for the available 16 fNIRS recordings, the two-stage motion artifacts removal technique, i.e., WPD-CCA has produced the best average (16.55 dB, utilizing db1 wavelet packet) and largest average (41.40%, using fk8 wavelet packet). The highest average and using single-stage artifacts removal techniques (WPD) are found as 16.11 dB and 26.40%, respectively, for all the fNIRS signals using fk4 wavelet packet. In both EEG and fNIRS modalities, the percentage reduction in motion artifacts increases by 11.28% and 56.82%, respectively when two-stage WPD-CCA techniques are employed in comparison with the single-stage WPD method. In addition, the average also increases when WPD-CCA techniques are used instead of single-stage WPD for both EEG and fNIRS signals. The increment in both and values is a clear indication that two-stage WPD-CCA performs relatively better compared to single-stage WPD. The results reported using the proposed methods outperform most of the existing state-of-the-art techniques.

Motion Artifacts Correction from Single-Channel EEG and fNIRS Signals Using Novel Wavelet Packet Decomposition in Combination with Canonical Correlation Analysis

The electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) signals, highly non-stationary in nature, greatly suffers from motion artifacts while recorded using wearable sensors. Since successful detection of various neurological and neuromuscular disorders is greatly dependent upon clean EEG and fNIRS signals, it is a matter of utmost importance to remove/reduce motion artifacts from EEG and fNIRS signals using reliable and robust methods. In this regard, this paper proposes two robust methods: (i) Wavelet packet decomposition (WPD) and (ii) WPD in combination with canonical correlation analysis (WPD-CCA), for motion artifact correction from single-channel EEG and fNIRS signals. The efficacy of these proposed techniques is tested using a benchmark dataset and the performance of the proposed methods is measured using two well-established performance matrices: (i) difference in the signal to noise ratio ( ) and (ii) percentage reduction in motion artifacts ( ). The proposed WPD-based single-stage motion artifacts correction technique produces the highest average (29.44 dB) when db2 wavelet packet is incorporated whereas the greatest average (53.48%) is obtained using db1 wavelet packet for all the available 23 EEG recordings. Our proposed two-stage motion artifacts correction technique, i.e., the WPD-CCA method utilizing db1 wavelet packet has shown the best denoising performance producing an average and values of 30.76 dB and 59.51%, respectively, for all the EEG recordings. On the other hand, for the available 16 fNIRS recordings, the two-stage motion artifacts removal technique, i.e., WPD-CCA has produced the best average (16.55 dB, utilizing db1 wavelet packet) and largest average (41.40%, using fk8 wavelet packet). The highest average and using single-stage artifacts removal techniques (WPD) are found as 16.11 dB and 26.40%, respectively, for all the fNIRS signals using fk4 wavelet packet. In both EEG and fNIRS modalities, the percentage reduction in motion artifacts increases by 11.28% and 56.82%, respectively when two-stage WPD-CCA techniques are employed in comparison with the single-stage WPD method. In addition, the average also increases when WPD-CCA techniques are used instead of single-stage WPD for both EEG and fNIRS signals. The increment in both and values is a clear indication that two-stage WPD-CCA performs relatively better compared to single-stage WPD. The results reported using the proposed methods outperform most of the existing state-of-the-art techniques.

Real-time recognition of different imagined actions on the same side of a single limb based on the fNIRS correlation coefficient

Functional near-infrared spectroscopy (fNIRS) is a type of functional brain imaging. Brain-computer interfaces (BCIs) based on fNIRS have recently been implemented. Most existing fNIRS-BCI studies have involved off-line analyses, but few studies used online performance testing. Furthermore, existing online fNIRS-BCI experimental paradigms have not yet carried out studies using different imagined movements of the same side of a single limb. In the present study, a real-time fNIRS-BCI system was constructed to identify two imagined movements of the same side of a single limb (right forearm and right hand). Ten healthy subjects were recruited and fNIRS signal was collected and real-time analyzed with two imagined movements (leftward movement involving the right forearm and right-hand clenching). In addition to the mean and slope features of fNIRS signals, the correlation coefficient between fNIRS signals induced by different imagined actions was extracted. A support vector machine (SVM) was used to classify the imagined actions. The average accuracy of real-time classification of the two imagined movements was 72.25 ± 0.004%. The findings suggest that different imagined movements on the same side of a single limb can be recognized real-time based on fNIRS, which may help to further guide the practical application of online fNIRS-BCIs.

Classification of Event-Related Potentials with Regularized Spatiotemporal LCMV Beamforming

The usability of EEG-based visual brain–computer interfaces (BCIs) based on event-related potentials (ERPs) benefits from reducing the calibration time before BCI operation. Linear decoding models, such as the spatiotemporal beamformer model, yield state-of-the-art accuracy. Although the training time of this model is generally low, it can require a substantial amount of training data to reach functional performance. Hence, BCI calibration sessions should be sufficiently long to provide enough training data. This work introduces two regularized estimators for the beamformer weights. The first estimator uses cross-validated L2-regularization. The second estimator exploits prior information about the structure of the EEG by assuming Kronecker–Toeplitz-structured covariance. The performances of these estimators are validated and compared with the original spatiotemporal beamformer and a Riemannian-geometry-based decoder using a BCI dataset with P300-paradigm recordings for 21 subjects. Our results show that the introduced estimators are well-conditioned in the presence of limited training data and improve ERP classification accuracy for unseen data. Additionally, we show that structured regularization results in lower training times and memory usage, and a more interpretable classification model.

Analyzing Classification Performance of fNIRS-BCI for Gait Rehabilitation Using Deep Neural Networks

This research presents a brain-computer interface (BCI) framework for brain signal classification using deep learning (DL) and machine learning (ML) approaches on functional near-infrared spectroscopy (fNIRS) signals. fNIRS signals of motor execution for walking and rest tasks are acquired from the primary motor cortex in the brain's left hemisphere for nine subjects. DL algorithms, including convolutional neural networks (CNNs), long short-term memory (LSTM), and bidirectional LSTM (Bi-LSTM) are used to achieve average classification accuracies of 88.50%, 84.24%, and 85.13%, respectively. For comparison purposes, three conventional ML algorithms, support vector machine (SVM), k-nearest neighbor (k-NN), and linear discriminant analysis (LDA) are also used for classification, resulting in average classification accuracies of 73.91%, 74.24%, and 65.85%, respectively. This study successfully demonstrates that the enhanced performance of fNIRS-BCI can be achieved in terms of classification accuracy using DL approaches compared to conventional ML approaches. Furthermore, the control commands generated by these classifiers can be used to initiate and stop the gait cycle of the lower limb exoskeleton for gait rehabilitation.

The brain state of motor imagery is reflected in the causal information of functional near-infrared spectroscopy

BACKGROUND

Brain-computer interface (BCI) is a promising neurorehabilitation strategy for ameliorating post-stroke function disorders. Physiological changes in the brain, such as functional near-infrared spectroscopy (fNIRS) dedicated to exploring cerebral circulatory responses during neurological rehabilitation tasks, are essential for gaining insights into neurorehabilitation mechanisms. However, the relationship between the neurovascular responses in different brain regions under rehabilitation tasks remains unknown.

OBJECTIVE

The present study explores the fNIRS interactions between brain regions under different motor imagery (MI) tasks, emphasizing functional characteristics of brain network patterns and BCI motor task classification.

METHODS

Granger causality analysis (GCA) is carried out for oxyhemoglobin data from 29 study participants in left- and right-hand MI tasks.

RESULTS

According to research findings, homozygous and heterozygous states in the two brain connectivity modes reveal one and nine channel pairs, respectively, with significantly different (P < 0.05) GC values under the left- and right-hand MI tasks in the population. With reference to the total 10 channel pairs of causality differences between the two brain working states, a support vector machine is used to classify the two tasks with an overall accuracy of 83% for five-fold cross-validation.

CONCLUSION

As demonstrated in the present study, fNIRS offers causality patterns in different brain states of MIBCI motor tasks. The research findings show that fNIRS causality can be used to assess different states of the brain, providing theoretical support for its application to neurorehabilitation assessment protocols to ultimately improve patients' quality of life.Video Abstract: http://links.lww.com/WNR/A653.

Brain-Computer Interfaces in Acute and Subacute Disorders of Consciousness

SUMMARY

Disorders of consciousness include coma, unresponsive wakefulness syndrome (also known as vegetative state), and minimally conscious state. Neurobehavioral scales such as coma recovery scale-revised are the gold standard for disorder of consciousness assessment. Brain-computer interfaces have been emerging as an alternative tool for these patients. The application of brain-computer interfaces in disorders of consciousness can be divided into four fields: assessment, communication, prediction, and rehabilitation. The operational theoretical model of consciousness that brain-computer interfaces explore was reviewed in this article, with a focus on studies with acute and subacute patients. We then proposed a clinically friendly guideline, which could contribute to the implementation of brain-computer interfaces in neurorehabilitation settings. Finally, we discussed limitations and future directions, including major challenges and possible solutions.

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

In this work, we introduce a deep learning architecture for evaluation on multimodal electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) recordings from 40 epileptic patients. Long short-term memory units and convolutional neural networks are integrated within a multimodal sequence-to-sequence autoencoder. The trained neural network predicts fNIRS signals from EEG, sans a priori, by hierarchically extracting deep features from EEG full spectra and specific EEG frequency bands. Results show that higher frequency EEG ranges are predictive of fNIRS signals with the gamma band inputs dominating fNIRS prediction as compared to other frequency envelopes. Seed based functional connectivity validates similar patterns between experimental fNIRS and our model's fNIRS reconstructions. This is the first study that shows it is possible to predict brain hemodynamics (fNIRS) from encoded neural data (EEG) in the resting human epileptic brain based on power spectrum amplitude modulation of frequency oscillations in the context of specific hypotheses about how EEG frequency bands decode fNIRS signals.

A 1D CNN for high accuracy classification and transfer learning in motor imagery EEG-based brain-computer interface

OBJECTIVE

Brain-computer interface (BCI) aims to establish communication paths between the brain processes and external devices. Different methods have been used to extract human intentions from electroencephalography (EEG) recordings. Those based on motor imagery (MI) seem to have a great potential for future applications. These approaches rely on the extraction of EEG distinctive patterns during imagined movements. Techniques able to extract patterns from raw signals represent an important target for BCI as they do not need labor-intensive data pre-processing.

APPROACH

We propose a new approach based on a 10-layer one-dimensional convolution neural network (1D-CNN) to classify five brain states (four MI classes plus a 'baseline' class) using a data augmentation algorithm and a limited number of EEG channels. In addition, we present a transfer learning method used to extract critical features from the EEG group dataset and then to customize the model to the single individual by training its outer layers with only 12-minute individual-related data.

MAIN RESULTS

The model tested with the 'EEG Motor Movement/Imagery Dataset' outperforms the current state-of-the-art models by achieving a 99.38% accuracy at the group level. In addition, the transfer learning approach we present achieves an average accuracy of 99.46%.

SIGNIFICANCE

The proposed methods could foster future BCI applications relying on few-channel portable recording devices and individual-based training.

Classification of Individual Finger Movements from Right Hand Using fNIRS Signals

Functional near-infrared spectroscopy (fNIRS) is a comparatively new noninvasive, portable, and easy-to-use brain imaging modality. However, complicated dexterous tasks such as individual finger-tapping, particularly using one hand, have been not investigated using fNIRS technology. Twenty-four healthy volunteers participated in the individual finger-tapping experiment. Data were acquired from the motor cortex using sixteen sources and sixteen detectors. In this preliminary study, we applied standard fNIRS data processing pipeline, i.e., optical densities conversation, signal processing, feature extraction, and classification algorithm implementation. Physiological and non-physiological noise is removed using 4th order band-pass Butter-worth and 3rd order Savitzky–Golay filters. Eight spatial statistical features were selected: signal-mean, peak, minimum, Skewness, Kurtosis, variance, median, and peak-to-peak form data of oxygenated haemoglobin changes. Sophisticated machine learning algorithms were applied, such as support vector machine (SVM), random forests (RF), decision trees (DT), AdaBoost, quadratic discriminant analysis (QDA), Artificial neural networks (ANN), k-nearest neighbors (kNN), and extreme gradient boosting (XGBoost). The average classification accuracies achieved were 0.75±0.04, 0.75±0.05, and 0.77±0.06 using k-nearest neighbors (kNN), Random forest (RF) and XGBoost, respectively. KNN, RF and XGBoost classifiers performed exceptionally well on such a high-class problem. The results need to be further investigated. In the future, a more in-depth analysis of the signal in both temporal and spatial domains will be conducted to investigate the underlying facts. The accuracies achieved are promising results and could open up a new research direction leading to enrichment of control commands generation for fNIRS-based brain-computer interface applications.

Comparison of functional activation responses from the auditory cortex derived using multi-distance frequency domain and continuous wave near-infrared spectroscopy

Significance: Quantitative measurements of cerebral hemodynamic changes due to functional activation are widely accomplished with commercial continuous wave (CW-NIRS) instruments despite the availability of the more rigorous multi-distance frequency domain (FD-NIRS) approach. A direct comparison of the two approaches to functional near-infrared spectroscopy can help in the interpretation of optical data and guide implementations of diffuse optical instruments for measuring functional activation. Aim: We explore the differences between CW-NIRS and multi-distance FD-NIRS by comparing measurements of functional activation in the human auditory cortex. Approach: Functional activation of the human auditory cortex was measured using a commercial frequency domain near-infrared spectroscopy instrument for 70 dB sound pressure level broadband noise and pure tone (1000 Hz) stimuli. Changes in tissue oxygenation were calculated using the modified Beer-Lambert law (CW-NIRS approach) and the photon diffusion equation (FD-NIRS approach). Results: Changes in oxygenated hemoglobin measured with the multi-distance FD-NIRS approach were about twice as large as those measured with the CW-NIRS approach. A finite-element simulation of the functional activation problem was performed to demonstrate that tissue oxygenation changes measured with the CW-NIRS approach is more accurate than that with multi-distance FD-NIRS. Conclusions: Multi-distance FD-NIRS approaches tend to overestimate functional activation effects, in part due to partial volume effects.

fNIRS Evidence for Distinguishing Patients With Major Depression and Healthy Controls

In recent years, major depressive disorder (MDD) has been shown to negatively impact physical recovery in a variety of patients. Functional near-infrared spectroscopy (fNIRS) is a tool that can potentially supplement clinical interviews and mental state examinations to establish a psychiatric diagnosis and monitor treatment progress. Thirty-two subjects, including 16 patients clinically diagnosed with MDD and 16 healthy controls (HCs), participated in the study. Brain oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) responses were recorded using a 22-channel continuous-wave fNIRS device while the subjects performed the emotional sound test. This study evaluated the difference between MDD patients and HCs using a variety of methods. In a comparison of the Pearson correlation coefficients between the HbO/HbR responses of each fNIRS channel and four scores, MDD patients and HCs had significantly different Athens Insomnia Scale (AIS) scores. By quantitative evaluation of the functional association, we found that MDD patients had aberrant functional connectivity compared with HCs. Furthermore, we concluded that compared with HCs, there were marked abnormalities in blood oxygen in the bilateral ventrolateral prefrontal cortex (VLPFC) and bilateral dorsolateral prefrontal cortex (DLPFC). Four statistical-based features extracted from HbO signals and four vector-based features from both HbO and HbR served as inputs to four simple neural networks (multilayer neural network (MNN), feedforward neural network (FNN), cascade forward neural network (CFNN) and recurrent neural network (RNN)). Through an analysis of combinations of different features, the combination of 4 common features (mean, STD, area under the receiver operating characteristic curve (AUC) and slope) yielded the highest classification accuracy of 89.74% for fear emotion. The combination of four novel feature (CBV, COE, |L | and K) resulted in a classification accuracy of 99.94% for fear emotion. The top 10 common and novel features were selected by the ReliefF feature selection algorithm, resulting in classification accuracies of 83.52% and 91.99%, respectively. This study identified the AUC and angle K as specific neuromarkers for predicting MDD across specific depression-related regions of the prefrontal cortex (PFC). These findings suggest that the fNIRS measurement of the PFC may serve as a supplementary test in routine clinical practice to further support a diagnosis of MDD.

Crossing time windows optimization based on mutual information for hybrid BCI

Hybrid EEG-fNIRS brain-computer interface (HBCI) is widely employed to enhance BCI performance. EEG and fNIRS signals are combined to increase the dimensionality of the information. Time windows are used to select EEG and fNIRS singles synchronously. However, it ignores that specific modal signals have their own characteristics, when the task is stimulated, the information between the modalities will mismatch at the moment, which has a significant impact on the classification performance. Here we propose a novel crossing time windows optimization for mental arithmetic (MA) based BCI. The EEG and fNIRS signals were segmented separately by sliding time windows. Then crossing time windows (CTW) were combined with each one segment from EEG and fNIRS selected independently. Furthermore, EEG and fNIRS features were extracted using Filter Bank Common Spatial Pattern (FBCSP) and statistical methods from each sample. Mutual information was calculated for FBCSP and statistical features to characterize the discrimination of crossing time windows, and the optimal window would be selected based on the largest mutual information. Finally, a sparse structured framework of Fisher Lasso feature selection (FLFS) was designed to select the joint features, and conventional Linear Discriminant Analysis (LDA) was employed to perform classification. We used proposed method for a MA dataset. The classification accuracy of the proposed method is 92.52 ± 5.38% and higher than other methods, which shows the rationality and superiority of the proposed method.

Optical Mapping of Brain Activity Underlying Directionality and Its Modulation by Expertise in Mandarin/English Interpreting

Recent neuroimaging research has suggested that unequal cognitive efforts exist between interpreting from language 1 (L1) to language 2 (L2) compared with interpreting from L2 to L1. However, the neural substrates that underlie this directionality effect are not yet well understood. Whether directionality is modulated by interpreting expertise also remains unknown. In this study, we recruited two groups of Mandarin (L1)/English (L2) bilingual speakers with varying levels of interpreting expertise and asked them to perform interpreting and reading tasks. Functional near-infrared spectroscopy (fNIRS) was used to collect cortical brain data for participants during each task, using 68 channels that covered the prefrontal cortex and the bilateral perisylvian regions. The interpreting-related neuroimaging data was normalized by using both L1 and L2 reading tasks, to control the function of reading and vocalization respectively. Our findings revealed the directionality effect in both groups, with forward interpreting (from L1 to L2) produced more pronounced brain activity, when normalized for reading. We also found that directionality was modulated by interpreting expertise in both normalizations. For the group with relatively high expertise, the activated brain regions included the right Broca's area and the left premotor and supplementary motor cortex; whereas for the group with relatively low expertise, the activated brain areas covered the superior temporal gyrus, the dorsolateral prefrontal cortex (DLPFC), the Broca's area, and visual area 3 in the right hemisphere. These findings indicated that interpreting expertise modulated brain activation, possibly because of more developed cognitive skills associated with executive functions in experienced interpreters.

Recognition of Flexion and Extension Imagery Involving the Right and Left Arms Based on Deep Belief Network and Functional Near-Infrared Spectroscopy

Brain-computer interaction based on motor imagery (MI) is an important brain-computer interface (BCI). Most methods for MI classification are based on electroencephalogram (EEG), and few studies have investigated signal processing based on MI-Functional Near-Infrared Spectroscopy (fNIRS). In addition, there is a need to improve the classification accuracy for MI fNIRS methods. In this study, a deep belief network (DBN) based on a restricted Boltzmann machine (RBM) was used to classify fNIRS signals of flexion and extension imagery involving the left and right arms. fNIRS signals from 16 channels covering the motor cortex area were recorded for each of 10 subjects executing or imagining flexion and extension involving the left and right arms. Oxygenated hemoglobin (HbO) concentration was used as a feature to train two RBMs that were subsequently stacked with an additional softmax regression output layer to construct DBN. We also explored the DBN model classification accuracy for the test dataset from one subject using training dataset from other subjects. The average DBN classification accuracy for flexion and extension movement and imagery involving the left and right arms was 84.35 ± 3.86% and 78.19 ± 3.73%, respectively. For a given DBN model, better classification results are obtained for test datasets for a given subject when the model is trained using dataset from the same subject than when the model is trained using datasets from other subjects. The results show that the DBN algorithm can effectively identify flexion and extension imagery involving the right and left arms using fNIRS. This study is expected to serve as a reference for constructing online MI-BCI systems based on DBN and fNIRS.

Correcting physiological noise in whole-head functional near-infrared spectroscopy

BACKGROUND

Functional near-infrared spectroscopy (fNIRS) has been increasingly employed to monitor cerebral hemodynamics in normal and diseased conditions. However, fNIRS suffers from its susceptibility to superficial activity and systemic physiological noise. The objective of the study was to establish a noise reduction method for fNIRS in a whole-head montage.

NEW METHOD

We have developed an automated denoising method for whole-head fNIRS. A high-density montage consisting of 109 long-separation channels and 8 short-separation channels was used for recording. Auxiliary sensors were also used to measure motion, respiration and pulse simultaneously. The method incorporates principal component analysis and general linear model to identify and remove a globally uniform superficial component. Our denoising method was evaluated in experimental data acquired from a group of healthy human subjects during a visually cued motor task and further compared with a minimal preprocessing method and three established denoising methods in the literature. Quantitative metrics including contrast-to-noise ratio, within-subject standard deviation and adjusted coefficient of determination were evaluated.

RESULTS

After denoising, whole-head topography of fNIRS revealed focal activations concurrently in the primary motor and visual areas.

COMPARISON WITH EXISTING METHODS

Analysis showed that our method improves upon the four established preprocessing methods in the literature.

CONCLUSIONS

An automatic, effective and robust preprocessing pipeline was established for removing physiological noise in whole-head fNIRS recordings. Our method can enable fNIRS as a reliable tool in monitoring large-scale, network-level brain activities for clinical uses.

Classification of Prefrontal Cortex Activity Based on Functional Near-Infrared Spectroscopy Data upon Olfactory Stimulation

The sense of smell is one of the most important organs in humans, and olfactory imaging can detect signals in the anterior orbital frontal lobe. This study assessed olfactory stimuli using support vector machines (SVMs) with signals from functional near-infrared spectroscopy (fNIRS) data obtained from the prefrontal cortex. These data included odor stimuli and air state, which triggered the hemodynamic response function (HRF), determined from variations in oxyhemoglobin (oxyHb) and deoxyhemoglobin (deoxyHb) levels; photoplethysmography (PPG) of two wavelengths (raw optical red and near-infrared data); and the ratios of data from two optical datasets. We adopted three SVM kernel functions (i.e., linear, quadratic, and cubic) to analyze signals and compare their performance with the HRF and PPG signals. The results revealed that oxyHb yielded the most efficient single-signal data with a quadratic kernel function, and a combination of HRF and PPG signals yielded the most efficient multi-signal data with the cubic function. Our results revealed superior SVM analysis of HRFs for classifying odor and air status using fNIRS data during olfaction in humans. Furthermore, the olfactory stimulation can be accurately classified by using quadratic and cubic kernel functions in SVM, even for an individual participant data set.

Research on fNIRS Recognition Method of Upper Limb Movement Intention

This paper aims at realizing upper limb rehabilitation training by using an fNIRS-BCI system. This article mainly focuses on the analysis and research of the cerebral blood oxygen signal in the system, and gradually extends the analysis and recognition method of the movement intention in the cerebral blood oxygen signal to the actual brain-computer interface system. Fifty subjects completed four upper limb movement paradigms: Lifting-up, putting down, pulling back, and pushing forward. Then, their near-infrared data and movement trigger signals were collected. In terms of the recognition algorithm for detecting the initial intention of upper limb movements, gradient boosting tree (GBDT) and random forest (RF) were selected for classification experiments. Finally, RF classifier with better comprehensive indicators was selected as the final classification algorithm. The best offline recognition rate was 94.4% (151/160). The ReliefF algorithm based on distance measurement and the genetic algorithm proposed in the genetic theory were used to select features. In terms of upper limb motion state recognition algorithms, logistic regression (LR), support vector machine (SVM), naive Bayes (NB), and linear discriminant analysis (LDA) were selected for experiments. Kappa coefficient was used as the classification index to evaluate the performance of the classifier. Finally, SVM classification got the best performance, and the four-class recognition accuracy rate was 84.4%. The results show that RF and SVM can achieve high recognition accuracy in motion intentions and the upper limb rehabilitation system designed in this paper has great application significance.

Classifying Motion Intention of Step Length and Synchronous Walking Speed by Functional Near-Infrared Spectroscopy

In some patients who have suffered an amputation or spinal cord injury, walking ability may be degraded or deteriorated. Helping these patients walk independently on their own initiative is of great significance. This paper proposes a method to identify subjects’ motion intention under different levels of step length and synchronous walking speed by using functional near-infrared spectroscopy technology. Thirty-one healthy subjects were recruited to walk under six given sets of gait parameters (small step with low/midspeed, midstep with low/mid/high speed, and large step with midspeed). The channels were subdivided into more regions. More frequency bands (6 subbands on average in the range of 0-0.18 Hz) were decomposed by applying the wavelet packet method. Further, a genetic algorithm and a library for support vector machine algorithm were applied for selecting typical feature vectors, which were represented by important regions with partial important channels mentioned above. The walking speed recognition rate was 71.21% in different step length states, and the step length recognition rate was 71.21% in different walking speed states. This study explores the method of identifying motion intention in two-dimensional multivariate states. It lays the foundation for controlling walking-assistance equipment adaptively based on cerebral hemoglobin information.

Multimodal fusion of EEG-fNIRS: a mutual information-based hybrid classification framework

Multimodal data fusion is one of the current primary neuroimaging research directions to overcome the fundamental limitations of individual modalities by exploiting complementary information from different modalities. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are especially compelling modalities due to their potentially complementary features reflecting the electro-hemodynamic characteristics of neural responses. However, the current multimodal studies lack a comprehensive systematic approach to properly merge the complementary features from their multimodal data. Identifying a systematic approach to properly fuse EEG-fNIRS data and exploit their complementary potential is crucial in improving performance. This paper proposes a framework for classifying fused EEG-fNIRS data at the feature level, relying on a mutual information-based feature selection approach with respect to the complementarity between features. The goal is to optimize the complementarity, redundancy and relevance between multimodal features with respect to the class labels as belonging to a pathological condition or healthy control. Nine amyotrophic lateral sclerosis (ALS) patients and nine controls underwent multimodal data recording during a visuo-mental task. Multiple spectral and temporal features were extracted and fed to a feature selection algorithm followed by a classifier, which selected the optimized subset of features through a cross-validation process. The results demonstrated considerably improved hybrid classification performance compared to the individual modalities and compared to conventional classification without feature selection, suggesting a potential efficacy of our proposed framework for wider neuro-clinical applications.

Decoding of Walking Imagery and Idle State Using Sparse Representation Based on fNIRS

Objectives

Brain-computer interface (BCI) based on functional near-infrared spectroscopy (fNIRS) is expected to provide an optional active rehabilitation training method for patients with walking dysfunction, which will affect their quality of life seriously. Sparse representation classification (SRC) oxyhemoglobin (HbO) concentration was used to decode walking imagery and idle state to construct fNIRS-BCI based on walking imagery.

Methods

15 subjects were recruited and fNIRS signals were collected during walking imagery and idle state. Firstly, band-pass filtering and baseline drift correction for HbO signal were carried out, and then the mean value, peak value, and root mean square (RMS) of HbO and their combinations were extracted as classification features; SRC was used to identify the extracted features and the result of SRC was compared with those of support vector machine (SVM), K-Nearest Neighbor (KNN), linear discriminant analysis (LDA), and logistic regression (LR).

Results

The experimental results showed that the average classification accuracy for walking imagery and idle state by SRC using three features combination was 91.55±3.30%, which was significantly higher than those of SVM, KNN, LDA, and LR (86.37±4.42%, 85.65±5.01%, 86.43±4.41%, and 76.14±5.32%, respectively), and the classification accuracy of other combined features was higher than that of single feature.

Conclusions

The study showed that introducing SRC into fNIRS-BCI can effectively identify walking imagery and idle state. It also showed that different time windows for feature extraction have an impact on the classification results, and the time window of 2–8 s achieved a better classification accuracy (94.33±2.60%) than other time windows. Significance. The study was expected to provide a new and optional active rehabilitation training method for patients with walking dysfunction. In addition, the experiment was also a rare study based on fNIRS-BCI using SRC to decode walking imagery and idle state.

Optical brain imaging and its application to neurofeedback

Besides passive recording of brain electric or magnetic activity, also non-ionizing electromagnetic or optical radiation can be used for real-time brain imaging. Here, changes in the radiation's absorption or scattering allow for continuous in vivo assessment of regional neurometabolic and neurovascular activity. Besides magnetic resonance imaging (MRI), over the last years, also functional near-infrared spectroscopy (fNIRS) was successfully established in real-time metabolic brain imaging. In contrast to MRI, fNIRS is portable and can be applied at bedside or in everyday life environments, e.g., to restore communication and movement. Here we provide a comprehensive overview of the history and state-of-the-art of real-time optical brain imaging with a special emphasis on its clinical use towards neurofeedback and brain-computer interface (BCI) applications. Besides pointing to the most critical challenges in clinical use, also novel approaches that combine real-time optical neuroimaging with other recording modalities (e.g. electro- or magnetoencephalography) are described, and their use in the context of neuroergonomics, neuroenhancement or neuroadaptive systems discussed.

Accurate hemodynamic response estimation by removal of stimulus-evoked superficial response in fNIRS signals

ObjectiveWe address the problem of hemodynamic response estimation when task-evoked extra-cerebral components are present in functional near-infrared spectroscopy (fNIRS) signals. These components might bias the hemodynamic response estimation, therefore careful and accurate denoising of data is needed.ApproachWe propose a dictionary-based algorithm to process every single event-related segment of the acquired signal for both long separation and short separation channels. Stimulus-evoked components and physiological noise are modeled by means of two distinct waveform dictionaries. For each segment, after removal of the physiological noise component in each channel, a template is employed to estimate stimulus-evoked responses in both channels. Then, the estimate from the short-separation channel is employed to correct for the evoked superficial response and refine the hemodynamic response estimate from the long-separation channel.Main resultsAnalysis of simulated, semi-simulated and real data shows that, by averaging single-segment estimates over multiple trials in an experiment, reliable results and improved accuracy compared to other methods can be obtained. The average estimation error of the proposed method for the semi-simulated data set is 34% for HbO and 78% for HbR, considering 40 trials. The proposed method outperforms the results of the methods proposed in the literature. While still far from the possibility of single-trial hemodynamic response estimation, a significant reduction in the number of averaged trials can also be obtained.SignificanceThis work proves that dedicated dictionaries can be successfully employed to model all different components of fNIRS signals. It demonstrates the effectiveness of a specifically designed algorithm structure in dealing with a complex denoising problem, enhancing the possibilities of fNIRS-based hemodynamic response analysis.

Decoding Three Different Preference Levels of Consumers Using Convolutional Neural Network: A Functional Near-Infrared Spectroscopy Study

This study decodes consumers' preference levels using a convolutional neural network (CNN) in neuromarketing. The classification accuracy in neuromarketing is a critical factor in evaluating the intentions of the consumers. Functional near-infrared spectroscopy (fNIRS) is utilized as a neuroimaging modality to measure the cerebral hemodynamic responses. In this study, a specific decoding structure, called CNN-based fNIRS-data analysis, was designed to achieve a high classification accuracy. Compared to other methods, the automated characteristics, constant training of the dataset, and learning efficiency of the proposed method are the main advantages. The experimental procedure required eight healthy participants (four female and four male) to view commercial advertisement videos of different durations (15, 30, and 60 s). The cerebral hemodynamic responses of the participants were measured. To compare the preference classification performances, CNN was utilized to extract the most common features, including the mean, peak, variance, kurtosis, and skewness. Considering three video durations, the average classification accuracies of 15, 30, and 60 s videos were 84.3, 87.9, and 86.4%, respectively. Among them, the classification accuracy of 87.9% for 30 s videos was the highest. The average classification accuracies of three preferences in females and males were 86.2 and 86.3%, respectively, showing no difference in each group. By comparing the classification performances in three different combinations (like vs. so-so, like vs. dislike, and so-so vs. dislike) between two groups, male participants were observed to have targeted preferences for commercial advertising, and the classification performance 88.4% between "like" vs. "dislike" out of three categories was the highest. Finally, pairwise classification performance are shown as follows: For female, 86.1% (like vs. so-so), 87.4% (like vs. dislike), 85.2% (so-so vs. dislike), and for male 85.7, 88.4, 85.1%, respectively.

Enhancing Classification Performance of fNIRS-BCI by Identifying Cortically Active Channels Using the z-Score Method

A state-of-the-art brain–computer interface (BCI) system includes brain signal acquisition, noise removal, channel selection, feature extraction, classification, and an application interface. In functional near-infrared spectroscopy-based BCI (fNIRS-BCI) channel selection may enhance classification performance by identifying suitable brain regions that contain brain activity. In this study, the z-score method for channel selection is proposed to improve fNIRS-BCI performance. The proposed method uses cross-correlation to match the similarity between desired and recorded brain activity signals, followed by forming a vector of each channel’s correlation coefficients’ maximum values. After that, the z-score is calculated for each value of that vector. A channel is selected based on a positive z-score value. The proposed method is applied to an open-access dataset containing mental arithmetic (MA) and motor imagery (MI) tasks for twenty-nine subjects. The proposed method is compared with the conventional t-value method and with no channel selected, i.e., using all channels. The z-score method yielded significantly improved (p < 0.0167) classification accuracies of 87.2 ± 7.0%, 88.4 ± 6.2%, and 88.1 ± 6.9% for left motor imagery (LMI) vs. rest, right motor imagery (RMI) vs. rest, and mental arithmetic (MA) vs. rest, respectively. The proposed method is also validated on an open-access database of 17 subjects, containing right-hand finger tapping (RFT), left-hand finger tapping (LFT), and dominant side foot tapping (FT) tasks.The study shows an enhanced performance of the z-score method over the t-value method as an advancement in efforts to improve state-of-the-art fNIRS-BCI systems’ performance.

An fNIRS-Based Motor Imagery BCI for ALS: A Subject-Specific Data-Driven Approach

OBJECTIVE

Functional near-infrared spectroscopy (fNIRS) has recently gained momentum in research on motor-imagery (MI)-based brain-computer interfaces (BCIs). However, strikingly, most of the research effort is primarily devoted to enhancing fNIRS-based BCIs for healthy individuals. The ability of patients with amyotrophic lateral sclerosis (ALS), among the main BCI end-users to utilize fNIRS-based hemodynamic responses to efficiently control an MI-based BCI, has not yet been explored. This study aims to quantify subject-specific spatio-temporal characteristics of ALS patients' hemodynamic responses to MI tasks, and to investigate the feasibility of using these responses as a means of communication to control a binary BCI.

METHODS

Hemodynamic responses were recorded using fNIRS from eight patients with ALS while performing MI-Rest tasks. The generalized linear model (GLM) analysis was conducted to statistically estimate and evaluate individualized spatial activation. Selected channel sets were statistically optimized for classification. Subject-specific discriminative features, including a proposed data-driven estimated coefficient obtained from GLM, and optimized classification parameters were identified and used to further evaluate the performance using a linear support vector machine (SVM) classifier.

RESULTS

Inter-subject variations were observed in spatio-temporal characteristics of patients' hemodynamic responses. Using optimized classification parameters and feature sets, all subjects could successfully use their MI hemodynamic responses to control a BCI with an average classification accuracy of 85.4% ± 9.8%.

SIGNIFICANCE

Our results indicate a promising application of fNIRS-based MI hemodynamic responses to control a binary BCI by ALS patients. These findings highlight the importance of subject-specific data-driven approaches for identifying discriminative spatio-temporal characteristics for an optimized BCI performance.

Performance Improvement for Detecting Brain Function Using fNIRS: A Multi-Distance Probe Configuration With PPL Method

To improve the spatial resolution of imaging and get more effective brain function information, a multi-distance probe configuration with three distances (28.2, 40, and 44.7 mm) and 52 channels is designed. At the same time, a data conversion method of modified Beer–Lambert law (MBLL) with partial pathlength (PPL) is proposed. In the experiment, three kinds of tasks, grip of left hand, grip of right hand, and rest, are performed with eight healthy subjects. First, with a typical single-distance probe configuration (30 mm, 24 channels), the feasibility of the proposed MBLL with PPL is preliminarily validated. Further, the characteristic of the proposed method is evaluated with the multi-distance probe configuration. Compared with MBLL with differential pathlength factor (DPF), the proposed MBLL with PPL is able to acquire more obvious concentration change and can achieve higher classification accuracy of the three tasks. Then, with the proposed method, the performance of the multi-distance probe configuration is discussed. Results show that, compared with a single distance, the combination of the three distances has better spatial resolution and could explore more accurate brain activation information. Besides, the classification accuracy of the three tasks obtained with the combination of three distances is higher than that of any combination of two distances. Also, with the combination of the three distances, the two-class classification between different tasks is carried out. Both theory and experimental results demonstrate that, using multi-distance probe configuration and the MBLL with PPL method, the performance of brain function detected by NIRS can be improved.

Sensor Location Optimization of Wireless Wearable fNIRS System for Cognitive Workload Monitoring Using a Data-Driven Approach for Improved Wearability

Functional Near-Infrared Spectroscopy (fNIRS) is a hemodynamic modality in human cognitive workload assessment receiving popularity due to its easier implementation, non-invasiveness, low cost and other benefits from the signal-processing point of view. Wearable wireless fNIRS systems used in research have promisingly shown that fNIRS could be used in cognitive workload assessment in out-of-the-lab scenarios, such as in operators' cognitive workload monitoring. In such a scenario, the wearability of the system is a significant factor affecting user comfort. In this respect, the wearability of the system can be improved if it is possible to minimize an fNIRS system without much compromise of the cognitive workload detection accuracy. In this study, cognitive workload-related hemodynamic changes were acquired using an fNIRS system covering the whole forehead, which is the region of interest in most cognitive workload-monitoring studies. A machine learning approach was applied to explore how the mean accuracy of the cognitive workload classification accuracy varied across various sensing locations on the forehead such as the Left, Mid, Right, Left-Mid, Right-Mid and Whole forehead. The statistical significance analysis result showed that the Mid location could result in significant cognitive workload classification accuracy compared to Whole forehead sensing, with a statistically insignificant difference in the mean accuracy. Thus, the wearable fNIRS system can be improved in terms of wearability by optimizing the sensor location, considering the sensing of the Mid location on the forehead for cognitive workload monitoring.

Enhanced Accuracy for Multiclass Mental Workload Detection Using Long Short-Term Memory for Brain-Computer Interface

Cognitive workload is one of the widely invoked human factors in the areas of human-machine interaction (HMI) and neuroergonomics. The precise assessment of cognitive and mental workload (MWL) is vital and requires accurate neuroimaging to monitor and evaluate the cognitive states of the brain. In this study, we have decoded four classes of MWL using long short-term memory (LSTM) with 89.31% average accuracy for brain-computer interface (BCI). The brain activity signals are acquired using functional near-infrared spectroscopy (fNIRS) from the prefrontal cortex (PFC) region of the brain. We performed a supervised MWL experimentation with four varying MWL levels on 15 participants (both male and female) and 10 trials of each MWL per participant. Real-time four-level MWL states are assessed using fNIRS system, and initial classification is performed using three strong machine learning (ML) techniques, support vector machine (SVM), k-nearest neighbor (k-NN), and artificial neural network (ANN) with obtained average accuracies of 54.33, 54.31, and 69.36%, respectively. In this study, novel deep learning (DL) frameworks are proposed, which utilizes convolutional neural network (CNN) and LSTM with 87.45 and 89.31% average accuracies, respectively, to solve high-dimensional four-level cognitive states classification problem. Statistical analysis, t-test, and one-way F-test (ANOVA) are also performed on accuracies obtained through ML and DL algorithms. Results show that the proposed DL (LSTM and CNN) algorithms significantly improve classification performance as compared with ML (SVM, ANN, and k-NN) algorithms.

Cortical Tasks-Based Optimal Filter Selection: An fNIRS Study

Functional near-infrared spectroscopy (fNIRS) is one of the latest noninvasive brain function measuring technique that has been used for the purpose of brain-computer interfacing (BCI). In this paper, we compare and analyze the effect of six most commonly used filtering techniques (i.e., Gaussian, Butterworth, Kalman, hemodynamic response filter (hrf), Wiener, and finite impulse response) on classification accuracies of fNIRS-BCI. To conclude with the best optimal filter for a specific cortical task owing to a specific cortical region, we divided our experimental tasks according to the three main cortical regions: prefrontal, motor, and visual cortex. Three different experiments were performed for prefrontal and motor execution tasks while one for visual stimuli. The tasks performed for prefrontal include rest (R) vs mental arithmetic (MA), R vs object rotation (OB), and OB vs MA. Similarly, for motor execution, R vs left finger tapping (LFT), R vs right finger tapping (RFT), and LFT vs RFT. Likewise, for the visual cortex, R vs visual stimuli (VS) task. These experiments were performed for ten trials with five subjects. For consistency among extracted data, six statistical features were evaluated using oxygenated hemoglobin, namely, slope, mean, peak, kurtosis, skewness, and variance. Combination of these six features was used to classify data by the nonlinear support vector machine (SVM). The classification accuracies obtained from SVM by using hrf and Gaussian were significantly higher for R vs MA, R vs OB, R vs RFT, and R vs VS and Wiener filter for OB vs MA. Similarly, for R vs LFT and LFT vs RFT, hrf was found to be significant p<0.05. These results show the feasibility of using hrf for effective removal of noises from fNIRS data.

Detection of Mild Cognitive Impairment Using Convolutional Neural Network: Temporal-Feature Maps of Functional Near-Infrared Spectroscopy

Mild cognitive impairment (MCI) is the clinical precursor of Alzheimer's disease (AD), which is considered the most common neurodegenerative disease in the elderly. Some MCI patients tend to remain stable over time and do not evolve to AD. It is essential to diagnose MCI in its early stages and provide timely treatment to the patient. In this study, we propose a neuroimaging approach to identify MCI using a deep learning method and functional near-infrared spectroscopy (fNIRS). For this purpose, fifteen MCI subjects and nine healthy controls (HCs) were asked to perform three mental tasks: N-back, Stroop, and verbal fluency (VF) tasks. Besides examining the oxygenated hemoglobin changes (ΔHbO) in the region of interest, ΔHbO maps at 13 specific time points (i.e., 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, and 65 s) during the tasks and seven temporal feature maps (i.e., two types of mean, three types of slope, kurtosis, and skewness) in the prefrontal cortex were investigated. A four-layer convolutional neural network (CNN) was applied to identify the subjects into either MCI or HC, individually, after training the CNN model with ΔHbO maps and temporal feature maps above. Finally, we used the 5-fold cross-validation approach to evaluate the performance of the CNN. The results of temporal feature maps exhibited high classification accuracies: The average accuracies for the N-back task, Stroop task, and VFT, respectively, were 89.46, 87.80, and 90.37%. Notably, the highest accuracy of 98.61% was achieved from the ΔHbO slope map during 20-60 s interval of N-back tasks. Our results indicate that the fNIRS imaging approach based on temporal feature maps is a promising diagnostic method for early detection of MCI and can be used as a tool for clinical doctors to identify MCI from their patients.