Single-trial classification of NIRS signals during emotional induction tasks: towards a corporeal machine interface

Investigating the relationship between CSAT scores and prefrontal fNIRS signals during cognitive tasks using a quantum annealing algorithm

Academic achievement is a critical measure of intellectual ability, prompting extensive research into cognitive tasks as potential predictors. Neuroimaging technologies, such as functional near-infrared spectroscopy (fNIRS), offer insights into brain hemodynamics, allowing understanding of the link between cognitive performance and academic achievement. Herein, we explored the association between cognitive tasks and academic achievement by analyzing prefrontal fNIRS signals. A novel quantum annealer (QA) feature selection algorithm was applied to fNIRS data to identify cognitive tasks correlated with CSAT scores. Twelve features (signal mean, median, variance, peak, number of peaks, sum of peaks, range, minimum, kurtosis, skewness, standard deviation, and root mean square) were extracted from fNIRS signals at two time windows (10- and 60-s) to compare results from various feature variable conditions. The feature selection results from the QA-based and XGBoost regressor algorithms were compared to validate the former's performance. In a two-step validation process using multiple linear regression models, model fitness (adjusted R2) and model prediction error (RMSE) values were calculated. The quantum annealer demonstrated comparable performance to classical machine learning models, and specific cognitive tasks, including verbal fluency, recognition, and the Corsi block tapping task, were correlated with academic achievement. Group analyses revealed stronger associations between Tower of London and N-back tasks with higher CSAT scores. Quantum annealing algorithms have significant potential in feature selection using fNIRS data, and represents a novel research approach. Future studies should explore predictors of academic achievement and cognitive ability.

Research on Emotion Recognition Method of Cerebral Blood Oxygen Signal Based on CNN-Transformer Network

In recent years, research on emotion recognition has become more and more popular, but there are few studies on emotion recognition based on cerebral blood oxygen signals. Since the electroencephalogram (EEG) is easily disturbed by eye movement and the portability is not high, this study uses a more comfortable and convenient functional near-infrared spectroscopy (fNIRS) system to record brain signals from participants while watching three different types of video clips. During the experiment, the changes in cerebral blood oxygen concentration in the 8 channels of the prefrontal cortex of the brain were collected and analyzed. We processed and divided the collected cerebral blood oxygen data, and used multiple classifiers to realize the identification of the three emotional states of joy, neutrality, and sadness. Since the classification accuracy of the convolutional neural network (CNN) in this research is not significantly superior to that of the XGBoost algorithm, this paper proposes a CNN-Transformer network based on the characteristics of time series data to improve the classification accuracy of ternary emotions. The network first uses convolution operations to extract channel features from multi-channel time series, then the features and the output information of the fully connected layer are input to the Transformer netork structure, and its multi-head attention mechanism is used to focus on different channel domain information, which has better spatiality. The experimental results show that the CNN-Transformer network can achieve 86.7% classification accuracy for ternary emotions, which is about 5% higher than the accuracy of CNN, and this provides some help for other research in the field of emotion recognition based on time series data such as fNIRS.

Explainable artificial intelligence model to predict brain states from fNIRS signals

Objective: Most Deep Learning (DL) methods for the classification of functional Near-Infrared Spectroscopy (fNIRS) signals do so without explaining which features contribute to the classification of a task or imagery. An explainable artificial intelligence (xAI) system that can decompose the Deep Learning mode's output onto the input variables for fNIRS signals is described here. Approach: We propose an xAI-fNIRS system that consists of a classification module and an explanation module. The classification module consists of two separately trained sliding window-based classifiers, namely, (i) 1-D Convolutional Neural Network (CNN); and (ii) Long Short-Term Memory (LSTM). The explanation module uses SHAP (SHapley Additive exPlanations) to explain the CNN model's output in terms of the model's input. Main results: We observed that the classification module was able to classify two types of datasets: (a) Motor task (MT), acquired from three subjects; and (b) Motor imagery (MI), acquired from 29 subjects, with an accuracy of over 96% for both CNN and LSTM models. The explanation module was able to identify the channels contributing the most to the classification of MI or MT and therefore identify the channel locations and whether they correspond to oxy- or deoxy-hemoglobin levels in those locations. Significance: The xAI-fNIRS system can distinguish between the brain states related to overt and covert motor imagery from fNIRS signals with high classification accuracy and is able to explain the signal features that discriminate between the brain states of interest.

Screening for Alzheimer's disease using prefrontal resting-state functional near-infrared spectroscopy

INTRODUCTION

Alzheimer's disease (AD) is neurodegenerative dementia that causes neurovascular dysfunction and cognitive impairment. Currently, 50 million people live with dementia worldwide, and there are nearly 10 million new cases every year. There is a need for relatively less costly and more objective methods of screening and early diagnosis.

METHODS

Functional near-infrared spectroscopy (fNIRS) systems are a promising solution for the early Detection of AD. For a practical clinically relevant system, a smaller number of optimally placed channels are clearly preferable. In this study, we investigated the number and locations of the best-performing fNIRS channels measuring prefrontal cortex activations. Twenty-one subjects diagnosed with AD and eighteen healthy controls were recruited for the study.

RESULTS

We have shown that resting-state fNIRS recordings from a small number of prefrontal locations provide a promising methodology for detecting AD and monitoring its progression. A high-density continuous-wave fNIRS system was first used to verify the relatively lower hemodynamic activity in the prefrontal cortical areas observed in patients with AD. By using the episode averaged standard deviation of the oxyhemoglobin concentration changes as features that were fed into a Support Vector Machine; we then showed that the accuracy of subsets of optical channels in predicting the presence and severity of AD was significantly above chance. The results suggest that AD can be detected with a 0.76 sensitivity score and a 0.68 specificity score while the severity of AD could be detected with a 0.75 sensitivity score and a 0.72 specificity score with ≤5 channels.

DISCUSSION

These scores suggest that fNIRS is a viable technology for conveniently detecting and monitoring AD as well as investigating underlying mechanisms of disease progression.

A pediatric near-infrared spectroscopy brain-computer interface based on the detection of emotional valence

Brain-computer interfaces (BCIs) are being investigated as an access pathway to communication for individuals with physical disabilities, as the technology obviates the need for voluntary motor control. However, to date, minimal research has investigated the use of BCIs for children. Traditional BCI communication paradigms may be suboptimal given that children with physical disabilities may face delays in cognitive development and acquisition of literacy skills. Instead, in this study we explored emotional state as an alternative access pathway to communication. We developed a pediatric BCI to identify positive and negative emotional states from changes in hemodynamic activity of the prefrontal cortex (PFC). To train and test the BCI, 10 neurotypical children aged 8–14 underwent a series of emotion-induction trials over four experimental sessions (one offline, three online) while their brain activity was measured with functional near-infrared spectroscopy (fNIRS). Visual neurofeedback was used to assist participants in regulating their emotional states and modulating their hemodynamic activity in response to the affective stimuli. Child-specific linear discriminant classifiers were trained on cumulatively available data from previous sessions and adaptively updated throughout each session. Average online valence classification exceeded chance across participants by the last two online sessions (with 7 and 8 of the 10 participants performing better than chance, respectively, in Sessions 3 and 4). There was a small significant positive correlation with online BCI performance and age, suggesting older participants were more successful at regulating their emotional state and/or brain activity. Variability was seen across participants in regards to BCI performance, hemodynamic response, and discriminatory features and channels. Retrospective offline analyses yielded accuracies comparable to those reported in adult affective BCI studies using fNIRS. Affective fNIRS-BCIs appear to be feasible for school-aged children, but to further gauge the practical potential of this type of BCI, replication with more training sessions, larger sample sizes, and end-users with disabilities is necessary.

Four-Class Classification of Neuropsychiatric Disorders by Use of Functional Near-Infrared Spectroscopy Derived Biomarkers

Diagnosis of most neuropsychiatric disorders relies on subjective measures, which makes the reliability of final clinical decisions questionable. The aim of this study was to propose a machine learning-based classification approach for objective diagnosis of three disorders of neuropsychiatric or neurological origin with functional near-infrared spectroscopy (fNIRS) derived biomarkers. Thirteen healthy adolescents and sixty-seven patients who were clinically diagnosed with migraine, obsessive compulsive disorder, or schizophrenia performed a Stroop task, while prefrontal cortex hemodynamics were monitored with fNIRS. Hemodynamic and cognitive features were extracted for training three supervised learning algorithms (naïve bayes (NB), linear discriminant analysis (LDA), and support vector machines (SVM)). The performance of each algorithm in correctly predicting the class of each participant across the four classes was tested with ten runs of a ten-fold cross-validation procedure. All algorithms achieved four-class classification performances with accuracies above 81% and specificities above 94%. SVM had the highest performance in terms of accuracy (85.1 ± 1.77%), sensitivity (84 ± 1.7%), specificity (95 ± 0.5%), precision (86 ± 1.6%), and F1-score (85 ± 1.7%). fNIRS-derived features have no subjective report bias when used for automated classification purposes. The presented methodology might have significant potential for assisting in the objective diagnosis of neuropsychiatric disorders associated with frontal lobe dysfunction.

The brain state of motor imagery is reflected in the causal information of functional near-infrared spectroscopy

BACKGROUND

Brain-computer interface (BCI) is a promising neurorehabilitation strategy for ameliorating post-stroke function disorders. Physiological changes in the brain, such as functional near-infrared spectroscopy (fNIRS) dedicated to exploring cerebral circulatory responses during neurological rehabilitation tasks, are essential for gaining insights into neurorehabilitation mechanisms. However, the relationship between the neurovascular responses in different brain regions under rehabilitation tasks remains unknown.

OBJECTIVE

The present study explores the fNIRS interactions between brain regions under different motor imagery (MI) tasks, emphasizing functional characteristics of brain network patterns and BCI motor task classification.

METHODS

Granger causality analysis (GCA) is carried out for oxyhemoglobin data from 29 study participants in left- and right-hand MI tasks.

RESULTS

According to research findings, homozygous and heterozygous states in the two brain connectivity modes reveal one and nine channel pairs, respectively, with significantly different (P < 0.05) GC values under the left- and right-hand MI tasks in the population. With reference to the total 10 channel pairs of causality differences between the two brain working states, a support vector machine is used to classify the two tasks with an overall accuracy of 83% for five-fold cross-validation.

CONCLUSION

As demonstrated in the present study, fNIRS offers causality patterns in different brain states of MIBCI motor tasks. The research findings show that fNIRS causality can be used to assess different states of the brain, providing theoretical support for its application to neurorehabilitation assessment protocols to ultimately improve patients' quality of life.Video Abstract: http://links.lww.com/WNR/A653.

Classification of fNIRS data with LDA and SVM: a proof-of-concept for application in infant studies

This study presents the implementation of a within-subject classification method, based on the use of Linear Discriminant Analysis (LDA) and Support Vector Machines (SVM), for the classification of hemodynamic responses. Using a synthetic dataset that closely resembles real experimental infant functional near-infrared spectroscopy (fNIRS) data, the impact of different levels of noise and different HRF amplitudes on the classification performances of the two classifiers are quantitively investigated.

A Chiral fNIRS Spotlight on Cerebellar Activation in a Finger Tapping Task

Functional Magnetic Resonance Imaging (fMRI) has been so far the golden standard to study the functional aspects of the cerebellum. In this paper, a low-cost alternative imaging, i.e. functional Near-Infrared Spectroscopy (fNIRS) is demonstrated to achieve successful measurements of the cerebellar hemodynamics towards the challenging observation of motor and cognitive processes at the cerebellar level. The excitation and reception optodes need to be properly placed to circumvent a major hindering from the shielding by the neck muscles. A simple experimental protocol, i.e. finger tapping task, was implemented to observe the subject's engagement and the presence of functional asymmetries. Marked differences among subjects with different levels of lateralization were clearly noticed in terms of activation and latencies, together with peaks in the hemodynamic response following neural activation. These preliminary results suggest also differences in the hemodynamic behavior between the brain and the cerebellum and encourage future and extended analysis in this direction.Clinical Relevance-This establishes the possibility to use a novel technique (fNIRS) to study cerebellar hemodynamics instead of fMRI.

Identification of impulsive adolescents with a functional near infrared spectroscopy (fNIRS) based decision support system

Background.The gold standard for diagnosing impulsivity relies on clinical interviews, behavioral questionnaires and rating scales which are highly subjective.Objective.The aim of this study was to develop a functional near infrared spectroscopy (fNIRS) based classification approach for correct identification of impulsive adolescents. Taking into account the multifaceted nature of impulsivity, we propose that combining informative features from clinical, behavioral and neurophysiological domains might better elucidate the neurobiological distinction underlying symptoms of impulsivity.Approach. Hemodynamic and behavioral information was collected from 38 impulsive adolescents and from 33 non-impulsive adolescents during a Stroop task with concurrent fNIRS recordings. Connectivity-based features were computed from the hemodynamic signals and a neural efficiency metric was computed by fusing the behavioral and connectivity-based features. We tested the efficacy of two commonly used supervised machine-learning methods, namely the support vector machines (SVM) and artificial neural networks (ANN) in discriminating impulsive adolescents from their non-impulsive peers when trained with multi-domain features. Wrapper method was adapted to identify the informative biomarkers in each domain. Classification accuracies of each algorithm were computed after 10 runs of a 10-fold cross-validation procedure, conducted for 7 different combinations of the 3-domain feature set.Main results.Both SVM and ANN achieved diagnostic accuracies above 90% when trained with Wrapper-selected clinical, behavioral and fNIRS derived features. SVM performed significantly higher than ANN in terms of the accuracy metric (92.2% and 90.16%, respectively,p= 0.005).Significance.Preliminary findings show the feasibility and applicability of both machine-learning based methods for correct identification of impulsive adolescents when trained with multi-domain data involving clinical interviews, fNIRS based biomarkers and neuropsychiatric test measures. The proposed automated classification approach holds promise for assisting the clinical practice of diagnosing impulsivity and other psychiatric disorders. Our results also pave the path for a computer-aided diagnosis perspective for rating the severity of impulsivity.

fNIRS Evidence for Distinguishing Patients With Major Depression and Healthy Controls

In recent years, major depressive disorder (MDD) has been shown to negatively impact physical recovery in a variety of patients. Functional near-infrared spectroscopy (fNIRS) is a tool that can potentially supplement clinical interviews and mental state examinations to establish a psychiatric diagnosis and monitor treatment progress. Thirty-two subjects, including 16 patients clinically diagnosed with MDD and 16 healthy controls (HCs), participated in the study. Brain oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) responses were recorded using a 22-channel continuous-wave fNIRS device while the subjects performed the emotional sound test. This study evaluated the difference between MDD patients and HCs using a variety of methods. In a comparison of the Pearson correlation coefficients between the HbO/HbR responses of each fNIRS channel and four scores, MDD patients and HCs had significantly different Athens Insomnia Scale (AIS) scores. By quantitative evaluation of the functional association, we found that MDD patients had aberrant functional connectivity compared with HCs. Furthermore, we concluded that compared with HCs, there were marked abnormalities in blood oxygen in the bilateral ventrolateral prefrontal cortex (VLPFC) and bilateral dorsolateral prefrontal cortex (DLPFC). Four statistical-based features extracted from HbO signals and four vector-based features from both HbO and HbR served as inputs to four simple neural networks (multilayer neural network (MNN), feedforward neural network (FNN), cascade forward neural network (CFNN) and recurrent neural network (RNN)). Through an analysis of combinations of different features, the combination of 4 common features (mean, STD, area under the receiver operating characteristic curve (AUC) and slope) yielded the highest classification accuracy of 89.74% for fear emotion. The combination of four novel feature (CBV, COE, |L | and K) resulted in a classification accuracy of 99.94% for fear emotion. The top 10 common and novel features were selected by the ReliefF feature selection algorithm, resulting in classification accuracies of 83.52% and 91.99%, respectively. This study identified the AUC and angle K as specific neuromarkers for predicting MDD across specific depression-related regions of the prefrontal cortex (PFC). These findings suggest that the fNIRS measurement of the PFC may serve as a supplementary test in routine clinical practice to further support a diagnosis of MDD.

Affective Neurofeedback Under Naturalistic Conditions: A Mini-Review of Current Achievements and Open Challenges

Affective neurofeedback training allows for the self-regulation of the putative circuits of emotion regulation. This approach has recently been studied as a possible additional treatment for psychiatric disorders, presenting positive effects in symptoms and behaviors. After neurofeedback training, a critical aspect is the transference of the learned self-regulation strategies to outside the laboratory and how to continue reinforcing these strategies in non-controlled environments. In this mini-review, we discuss the current achievements of affective neurofeedback under naturalistic setups. For this, we first provide a brief overview of the state-of-the-art for affective neurofeedback protocols. We then discuss virtual reality as a transitional step toward the final goal of "in-the-wild" protocols and current advances using mobile neurotechnology. Finally, we provide a discussion of open challenges for affective neurofeedback protocols in-the-wild, including topics such as convenience and reliability, environmental effects in attention and workload, among others.

Decoding Multiple Sound-Categories in the Auditory Cortex by Neural Networks: An fNIRS Study

This study aims to decode the hemodynamic responses (HRs) evoked by multiple sound-categories using functional near-infrared spectroscopy (fNIRS). The six different sounds were given as stimuli (English, non-English, annoying, nature, music, and gunshot). The oxy-hemoglobin (HbO) concentration changes are measured in both hemispheres of the auditory cortex while 18 healthy subjects listen to 10-s blocks of six sound-categories. Long short-term memory (LSTM) networks were used as a classifier. The classification accuracy was 20.38 ± 4.63% with six class classification. Though LSTM networks' performance was a little higher than chance levels, it is noteworthy that we could classify the data subject-wise without feature selections.

Closed-loop neurostimulation for affective symptoms and disorders: An overview

Affective and anxiety disorders are the most prevalent and incident psychiatric disorders worldwide. Therapeutic approaches to these disorders using non-invasive brain stimulation (NIBS) and analogous techniques have been extensively investigated. In this paper, we discuss the combination of NIBS and neurofeedback in closed-loop setups and its application for affective symptoms and disorders. For this, we first provide a rationale for this combination by presenting some of the main original findings of NIBS, with a primary focus on transcranial magnetic stimulation (TMS), and neurofeedback, including protocols based on electroencephalography (EEG) and functional magnetic resonance imaging (fMRI). Then, we provide a scope review of studies combining real-time neurofeedback with NIBS protocols in the so-called closed-loop brain state-dependent neuromodulation (BSDS). Finally, we discuss the concomitant use of TMS and real-time functional near-infrared spectroscopy (fNIRS) as a possible solution to the current limitations of BSDS-based protocols for affective and anxiety disorders.

High density optical neuroimaging predicts surgeons's subjective experience and skill levels

Measuring cognitive load is important for surgical education and patient safety. Traditional approaches of measuring cognitive load of surgeons utilise behavioural metrics to measure performance and surveys and questionnaires to collect reports of subjective experience. These have disadvantages such as sporadic data, occasionally intrusive methodologies, subjective or misleading self-reporting. In addition, traditional approaches use subjective metrics that cannot distinguish between skill levels. Functional neuroimaging data was collected using a high density, wireless NIRS device from sixteen surgeons (11 attending surgeons and 5 surgery resident) and 17 students while they performed two laparoscopic tasks (Peg transfer and String pass). Participant’s subjective mental load was assessed using the NASA-TLX survey. Machine learning approaches were used for predicting the subjective experience and skill levels. The Prefrontal cortex (PFC) activations were greater in students who reported higher-than-median task load, as measured by the NASA-TLX survey. However in the case of attending surgeons the opposite tendency was observed, namely higher activations in the lower v higher task loaded subjects. We found that response was greater in the left PFC of students particularly near the dorso- and ventrolateral areas. We quantified the ability of PFC activation to predict the differences in skill and task load using machine learning while focussing on the effects of NIRS channel separation distance on the results. Our results showed that the classification of skill level and subjective task load could be predicted based on PFC activation with an accuracy of nearly 90%. Our finding shows that there is sufficient information available in the optical signals to make accurate predictions about the surgeons’ subjective experiences and skill levels. The high accuracy of results is encouraging and suggest the integration of the strategy developed in this study as a promising approach to design automated, more accurate and objective evaluation methods.

Subject-independent decoding of affective states using functional near-infrared spectroscopy

Affective decoding is the inference of human emotional states using brain signal measurements. This approach is crucial to develop new therapeutic approaches for psychiatric rehabilitation, such as affective neurofeedback protocols. To reduce the training duration and optimize the clinical outputs, an ideal clinical neurofeedback could be trained using data from an independent group of volunteers before being used by new patients. Here, we investigated if this subject-independent design of affective decoding can be achieved using functional near-infrared spectroscopy (fNIRS) signals from frontal and occipital areas. For this purpose, a linear discriminant analysis classifier was first trained in a dataset (49 participants, 24.65±3.23 years) and then tested in a completely independent one (20 participants, 24.00±3.92 years). Significant balanced accuracies between classes were found for positive vs. negative (64.50 ± 12.03%, p<0.01) and negative vs. neutral (68.25 ± 12.97%, p<0.01) affective states discrimination during a reactive block consisting in viewing affective-loaded images. For an active block, in which volunteers were instructed to recollect personal affective experiences, significant accuracy was found for positive vs. neutral affect classification (71.25 ± 18.02%, p<0.01). In this last case, only three fNIRS channels were enough to discriminate between neutral and positive affective states. Although more research is needed, for example focusing on better combinations of features and classifiers, our results highlight fNIRS as a possible technique for subject-independent affective decoding, reaching significant classification accuracies of emotional states using only a few but biologically relevant features.

Subject-Specific feature selection for near infrared spectroscopy based brain-computer interfaces

BACKGROUND AND OBJECTIVE

Brain-computer interfaces (BCIs) enable people to control an external device by analyzing the brain's neural activity. Functional near-infrared spectroscopy (fNIRS), which is an emerging optical imaging technique, is frequently used in non-invasive BCIs. Determining the subject-specific features is an important concern in enhancing the classification accuracy as well as reducing the complexity of fNIRS based BCI systems. In this study, the effectiveness of subject-specific feature selection on classification accuracy of fNIRS signals is examined.

METHODS

In order to determine the subject-specific optimal feature subsets, stepwise regression analysis based on sequential feature selection (SWR-SFS) and ReliefF methods were employed. Feature selection is applied on time-domain features of fNIRS signals such as mean, slope, peak, skewness and kurtosis values of signals. Linear discriminant analysis, k nearest neighborhood and support vector machines are employed to evaluate the performance of the selected feature subsets. The proposed techniques are validated on benchmark motor imagery (MI) and mental arithmetic (MA) based fNIRS datasets collected from 29 healthy subjects.

RESULTS

Both SWR-SFS and reliefF feature selection methods have significantly improved the classification accuracy. However, the best results (88.67% (HbR) and 86.43% (HbO) for MA dataset and 77.01% (HbR) and 71.32% (HbO) for MI dataset) were achieved using SWR-SFS while feature selection provided extremely high feature reduction rates (89.50% (HbR) and 93.99% (HbO) for MA dataset and 94.04% (HbR) and 97.73% (HbO) for MI dataset).

CONCLUSIONS

The results of the study indicate that employing feature selection improves both MA and MI-based fNIRS signals classification performance significantly.

Enhanced Performance of a Brain Switch by Simultaneous Use of EEG and NIRS Data for Asynchronous Brain-Computer Interface

Previous studies have shown the superior performance of hybrid electroencephalography (EEG)/ near-infrared spectroscopy (NIRS) brain-computer interfaces (BCIs). However, it has been veiled whether the use of a hybrid EEG/NIRS modality can provide better performance for a brain switch that can detect the onset of the intention to turn on a BCI. In this study, we developed such a hybrid EEG/NIRS brain switch and compared its performance with single modality EEG- and NIRS-based brain switch respectively, in terms of true positive rate (TPR), false positive rate (FPR), onset detection time (ODT), and information transfer rate (ITR). In an offline analysis, the performance of a hybrid EEG/NIRS brain switch was significantly improved over that of EEG- and NIRS-based brain switches in general, and in particular a significantly lower FPR was observed for the hybrid EEG/NIRS brain switch. A pseudo-online analysis was additionally performed to confirm the feasibility of implementing an online BCI system with our hybrid EEG/NIRS brain switch. The overall trend of pseudo-online analysis results generally coincided with that of the offline analysis results. No significant difference in all performance measures was also found between offline and pseudo online analysis schemes when the amount of training data was same, with one exception for the ITRs of an EEG brain switch. These offline and pseudo-online results demonstrate that a hybrid EEG/NIRS brain switch can be used to provide a better onset detection performance than that of a single neuroimaging modality.

Enhanced Accuracy for Multiclass Mental Workload Detection Using Long Short-Term Memory for Brain-Computer Interface

Cognitive workload is one of the widely invoked human factors in the areas of human-machine interaction (HMI) and neuroergonomics. The precise assessment of cognitive and mental workload (MWL) is vital and requires accurate neuroimaging to monitor and evaluate the cognitive states of the brain. In this study, we have decoded four classes of MWL using long short-term memory (LSTM) with 89.31% average accuracy for brain-computer interface (BCI). The brain activity signals are acquired using functional near-infrared spectroscopy (fNIRS) from the prefrontal cortex (PFC) region of the brain. We performed a supervised MWL experimentation with four varying MWL levels on 15 participants (both male and female) and 10 trials of each MWL per participant. Real-time four-level MWL states are assessed using fNIRS system, and initial classification is performed using three strong machine learning (ML) techniques, support vector machine (SVM), k-nearest neighbor (k-NN), and artificial neural network (ANN) with obtained average accuracies of 54.33, 54.31, and 69.36%, respectively. In this study, novel deep learning (DL) frameworks are proposed, which utilizes convolutional neural network (CNN) and LSTM with 87.45 and 89.31% average accuracies, respectively, to solve high-dimensional four-level cognitive states classification problem. Statistical analysis, t-test, and one-way F-test (ANOVA) are also performed on accuracies obtained through ML and DL algorithms. Results show that the proposed DL (LSTM and CNN) algorithms significantly improve classification performance as compared with ML (SVM, ANN, and k-NN) algorithms.

Tensor-Based Emotional Category Classification via Visual Attention-Based Heterogeneous CNN Feature Fusion

The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified.

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Human facial expressions are regarded as a vital indicator of one's emotion and intention, and even reveal the state of health and wellbeing. Emotional states have been associated with information processing within and between subcortical and cortical areas of the brain, including the amygdala and prefrontal cortex. In this study, we evaluated the relationship between spontaneous human facial affective expressions and multi-modal brain activity measured via non-invasive and wearable sensors: functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) signals. The affective states of twelve male participants detected via fNIRS, EEG, and spontaneous facial expressions were investigated in response to both image-content stimuli and video-content stimuli. We propose a method to jointly evaluate fNIRS and EEG signals for affective state detection (emotional valence as positive or negative). Experimental results reveal a strong correlation between spontaneous facial affective expressions and the perceived emotional valence. Moreover, the affective states were estimated by the fNIRS, EEG, and fNIRS + EEG brain activity measurements. We show that the proposed EEG + fNIRS hybrid method outperforms fNIRS-only and EEG-only approaches. Our findings indicate that the dynamic (video-content based) stimuli triggers a larger affective response than the static (image-content based) stimuli. These findings also suggest joint utilization of facial expression and wearable neuroimaging, fNIRS, and EEG, for improved emotional analysis and affective brain-computer interface applications.

Intersession Instability in fNIRS-Based Emotion Recognition

Emotion recognition based on neural signals is a promising technique for the detection of patients' emotions for enhancing healthcare. However, emotion-related neural signals, such as from functional near infrared spectroscopy (fNIRS), can be affected by various psychophysiological and environmental factors. There is a paucity of literature regarding data instability and classification instability in fNIRS-based emotion recognition systems, phenomenon which may lead to user dissatisfaction and abandonment. We collected data in an fNIRS-based 2-class emotion recognition test-retest experiment (3 week interval) with visual stimuli emotion induction to examine data instability and its impact on classification accuracy. We found a 22.2% average deterioration of emotion classification accuracy between the two sessions, suggesting that classification instability is a serious problem. We found that the changes in the distributions of the selected neural signal features, as evaluated by Kullback-Leibler (KL) divergence, were a likely cause of the accuracy decline. We analyzed the data instability and our results showed that instability of spatial activation patterns and instability of the hemodynamic response in the most activated region are correlated with accuracy decline. Finally, we propose a method for mitigating classification instability in fNIRS-based emotion recognition based on feature selection for stable features, the first such method to our knowledge. This new feature selection criterion considers not only the separability of features (evaluated by Fisher Score) but also their stability over time (evaluated by KL divergence between feature distributions at different time points). Testing showed that this method led to an approximately 5% improvement in cross-session generalization accuracy.

fNIRS Evidence for Recognizably Different Positive Emotions

The behavioral differentiation of positive emotions has recently been studied in terms of their discrete adaptive functions or appraising profiles. Some preliminary neurophysiological evidences have been found with electroencephalography or autonomic nervous system measurements such as heart rate, skin conductance, etc. However, the brain's hemodynamic responses to different positive emotions remain largely unknown. In the present study, the functional near-infrared spectroscopy (fNIRS) technique was employed. With this tool, we for the first time reported recognizable discrete positive emotions using fNIRS signals. Thirteen participants watched 30 emotional video clips to elicit 10 typical kinds of positive emotions (joy, gratitude, serenity, interest, hope, pride, amusement, inspiration, awe, and love), and their frontal neural activities were simultaneously recorded with a 24-channel fNIRS system. The multidimensional scaling analysis of participants' subjective ratings on these 10 positive emotions revealed three distinct clusters, which could be interpreted as "playfulness" for amusement, joy, interest, "encouragement" for awe, gratitude, hope, inspiration, pride, and "harmony" for love, serenity. Hemodynamic responses to these three positive emotion clusters showed distinct patterns, and HbO-based individual-level binary classifications between them achieved an averaged accuracy of 73.79 ± 11.49% (77.56 ± 7.39% for encouragement vs. harmony, 73.29 ± 11.87% for playfulness vs. harmony, 70.51 ± 13.96% for encouragement vs. harmony). Benefited from fNIRS's high portability, low running cost and the relative robustness against motion and electrical artifacts, our findings provided support for implementing a more fine-grained emotion recognition system with subdivided positive emotion categories.

Classification of motor imagery and execution signals with population-level feature sets: implications for probe design in fNIRS based BCI

OBJECTIVE

The aim of this study was to introduce a novel methodology for classification of brain hemodynamic responses collected via functional near infrared spectroscopy (fNIRS) during rest, motor imagery (MI) and motor execution (ME) tasks which involves generating population-level training sets.

APPROACH

A 48-channel fNIRS system was utilized to obtain hemodynamic signals from the frontal (FC), primary motor (PMC) and somatosensory cortex (SMC) of ten subjects during an experimental paradigm consisting of MI and ME of various right hand movements. Classification accuracies of random forest (RF), support vector machines (SVM), and artificial neural networks (ANN) were computed at the single subject level by training each classifier with subject specific features, and at the group level by training with features from all subjects for ME versus Rest, MI versus Rest and MI versus ME conditions. The performances were also computed for channel data restricted to FC, PMC and SMC regions separately to determine optimal probe location.

MAIN RESULTS

RF, SVM and ANN had comparably high classification accuracies for ME versus Rest (%94, %96 and %98 respectively) and for MI versus Rest (%95, %95 and %98 respectively) when fed with group level feature sets. The accuracy performance of each algorithm in localized brain regions were comparable (>%93) to the accuracy performance obtained with whole brain channels (>%94) for both ME versus Rest and MI versus Rest conditions.

SIGNIFICANCE

By demonstrating the feasibility of generating a population level training set with a high classification performance for three different classification algorithms, the findings pave the path for removing the necessity to acquire subject specific training data and hold promise for a novel, real-time fNIRS based BCI system design which will be most effective for application to disease populations for whom obtaining data to train a classification algorithm is not possible.

An fNIRS-based investigation of visual merchandising displays for fashion stores

This paper investigates a brain-based approach for visual merchandising display (VMD) in fashion stores. In marketing, VMD has become a research topic of interest. However, VMD research using brain activation information is rare. We examine the hemodynamic responses (HRs) in the prefrontal cortex (PFC) using functional near-infrared spectroscopy (fNIRS) while positive/negative displays of four stores (menswear, womenswear, underwear, and sportswear) are shown to 20 subjects. As features for classifying the HRs, the mean, variance, peak, skewness, kurtosis, t-value, and slope of the signals for a 20-sec time window for the activated channels are analyzed. Linear discriminant analysis is used for classifying two-class (positive and negative displays) and four-class (four fashion stores) models. PFC brain activation maps based on t-values depicting the data from the 16 channels are provided. In the two-class classification, the underwear store had the highest average classification result of 67.04%, whereas the menswear store had the lowest value of 64.15%. Men's classification accuracy for the underwear stores with positive and negative displays was 71.38%, whereas the highest classification accuracy obtained by women for womenswear stores was 73%. The average accuracy over the 20 subjects for positive displays was 50.68%, while that of negative displays was 51.07%. Therefore, these findings suggest that human brain activation is involved in the evaluation of the fashion store displays. It is concluded that fNIRS can be used as a brain-based tool in the evaluation of fashion stores in a daily life environment.

Feature Extraction and Classification Methods for Hybrid fNIRS-EEG Brain-Computer Interfaces

In this study, a brain-computer interface (BCI) framework for hybrid functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) for locked-in syndrome (LIS) patients is investigated. Brain tasks, channel selection methods, and feature extraction and classification algorithms available in the literature are reviewed. First, we categorize various types of patients with cognitive and motor impairments to assess the suitability of BCI for each of them. The prefrontal cortex is identified as a suitable brain region for imaging. Second, the brain activity that contributes to the generation of hemodynamic signals is reviewed. Mental arithmetic and word formation tasks are found to be suitable for use with LIS patients. Third, since a specific targeted brain region is needed for BCI, methods for determining the region of interest are reviewed. The combination of a bundled-optode configuration and threshold-integrated vector phase analysis turns out to be a promising solution. Fourth, the usable fNIRS features and EEG features are reviewed. For hybrid BCI, a combination of the signal peak and mean fNIRS signals and the highest band powers of EEG signals is promising. For classification, linear discriminant analysis has been most widely used. However, further research on vector phase analysis as a classifier for multiple commands is desirable. Overall, proper brain region identification and proper selection of features will improve classification accuracy. In conclusion, five future research issues are identified, and a new BCI scheme, including brain therapy for LIS patients and using the framework of hybrid fNIRS-EEG BCI, is provided.

Improvement of Information Transfer Rates Using a Hybrid EEG-NIRS Brain-Computer Interface with a Short Trial Length: Offline and Pseudo-Online Analyses

Electroencephalography (EEG) and near-infrared spectroscopy (NIRS) are non-invasive neuroimaging methods that record the electrical and metabolic activity of the brain, respectively. Hybrid EEG-NIRS brain-computer interfaces (hBCIs) that use complementary EEG and NIRS information to enhance BCI performance have recently emerged to overcome the limitations of existing unimodal BCIs, such as vulnerability to motion artifacts for EEG-BCI or low temporal resolution for NIRS-BCI. However, with respect to NIRS-BCI, in order to fully induce a task-related brain activation, a relatively long trial length (≥10 s) is selected owing to the inherent hemodynamic delay that lowers the information transfer rate (ITR; bits/min). To alleviate the ITR degradation, we propose a more practical hBCI operated by intuitive mental tasks, such as mental arithmetic (MA) and word chain (WC) tasks, performed within a short trial length (5 s). In addition, the suitability of the WC as a BCI task was assessed, which has so far rarely been used in the BCI field. In this experiment, EEG and NIRS data were simultaneously recorded while participants performed MA and WC tasks without preliminary training and remained relaxed (baseline; BL). Each task was performed for 5 s, which was a shorter time than previous hBCI studies. Subsequently, a classification was performed to discriminate MA-related or WC-related brain activations from BL-related activations. By using hBCI in the offline/pseudo-online analyses, average classification accuracies of 90.0 ± 7.1/85.5 ± 8.1% and 85.8 ± 8.6/79.5 ± 13.4% for MA vs. BL and WC vs. BL, respectively, were achieved. These were significantly higher than those of the unimodal EEG- or NIRS-BCI in most cases. Given the short trial length and improved classification accuracy, the average ITRs were improved by more than 96.6% for MA vs. BL and 87.1% for WC vs. BL, respectively, compared to those reported in previous studies. The suitability of implementing a more practical hBCI based on intuitive mental tasks without preliminary training and with a shorter trial length was validated when compared to previous studies.

Wavelet Transform Analysis the Recognizing Brain Activities for Development the Palm-Size and Simplification Near-Infrared Spectroscopy Prototype System by Using Arduino

Functional near-infrared spectroscopy (fNIRS) and brain computer interface (BCI) have become indispensable tools for recording and monitoring brain activity, comprising a non-invasive and safe technique that allows researchers to monitor blood flow in the front part of the brain. Although some medical device manufacturers developed complex fNIRS systems, downsized fNIRS systems are important for other uses, such as in portable (palm-sized) and wearable healthcare devices. This paper proposes a downsized compact fNIRS prototype that detects hemodynamics in the frontal lobe. The aim is to develop a compact fNIRS system, which is reliable and easy to integrate into portable (palm-sized) BCI devices. Through practical experiments with human subjects, our proposed system showed an ability to detect and monitor the start and end time of human brain activities when participants were solving a calculation table.

In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic work and everyday life settings. The emerging field of research, known as neuroergonomics, promotes the use of wearable and portable brain monitoring sensors such as functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. The objective of this study was to implement an on-line passive fNIRS-based brain computer interface to discriminate two levels of working memory load during highly ecological aircraft piloting tasks. Twenty eight recruited pilots were equally split into two groups (flight simulator vs. real aircraft). In both cases, identical approaches and experimental stimuli were used (serial memorization task, consisting in repeating series of pre-recorded air traffic control instructions, easy vs. hard). The results show pilots in the real flight condition committed more errors and had higher anterior prefrontal cortex activation than pilots in the simulator, when completing cognitively demanding tasks. Nevertheless, evaluation of single trial working memory load classification showed high accuracy (>76%) across both experimental conditions. The contributions here are two-fold. First, we demonstrate the feasibility of passively monitoring cognitive load in a realistic and complex situation (live piloting of an aircraft). In addition, the differences in performance and brain activity between the two experimental conditions underscore the need for ecologically-valid investigations.

Assessment of user voluntary engagement during neurorehabilitation using functional near-infrared spectroscopy: a preliminary study

BACKGROUND

Functional near infrared spectroscopy (fNIRS) finds extended applications in a variety of neuroscience fields. We investigated the potential of fNIRS to monitor voluntary engagement of users during neurorehabilitation, especially during combinatory exercise (CE) that simultaneously uses both, passive and active exercises. Although the CE approach can enhance neurorehabilitation outcome, compared to the conventional passive or active exercise strategies, the active engagement of patients in active motor movements during CE is not known.

METHODS

We determined hemodynamic responses induced by passive exercise and CE to evaluate the active involvement of users during CEs using fNIRS. In this preliminary study, hemodynamic responses of eight healthy subjects during three different tasks (passive exercise alone, passive exercise with motor imagery, and passive exercise with active motor execution) were recorded. On obtaining statistically significant differences, we classified the hemodynamic responses induced by passive exercise and CEs to determine the identification accuracy of the voluntary engagement of users using fNIRS.

RESULTS

Stronger and broader activation around the sensorimotor cortex was observed during CEs, compared to that during passive exercise. Moreover, pattern classification results revealed more than 80% accuracy.

CONCLUSIONS

Our preliminary study demonstrated that fNIRS can be potentially used to assess the engagement of users of the combinatory neurorehabilitation strategy.

fNIRS-based Neurorobotic Interface for gait rehabilitation

BACKGROUND

In this paper, a novel functional near-infrared spectroscopy (fNIRS)-based brain-computer interface (BCI) framework for control of prosthetic legs and rehabilitation of patients suffering from locomotive disorders is presented.

METHODS

fNIRS signals are used to initiate and stop the gait cycle, while a nonlinear proportional derivative computed torque controller (PD-CTC) with gravity compensation is used to control the torques of hip and knee joints for minimization of position error. In the present study, the brain signals of walking intention and rest tasks were acquired from the left hemisphere's primary motor cortex for nine subjects. Thereafter, for removal of motion artifacts and physiological noises, the performances of six different filters (i.e. Kalman, Wiener, Gaussian, hemodynamic response filter (hrf), Band-pass, finite impulse response) were evaluated. Then, six different features were extracted from oxygenated hemoglobin signals, and their different combinations were used for classification. Also, the classification performances of five different classifiers (i.e. k-Nearest Neighbour, quadratic discriminant analysis, linear discriminant analysis (LDA), Naïve Bayes, support vector machine (SVM)) were tested.

RESULTS

The classification accuracies obtained from SVM using the hrf were significantly higher (p < 0.01) than those of the other classifier/ filter combinations. Those accuracies were 77.5, 72.5, 68.3, 74.2, 73.3, 80.8, 65, 76.7, and 86.7% for the nine subjects, respectively.

CONCLUSION

The control commands generated using the classifiers initiated and stopped the gait cycle of the prosthetic leg, the knee and hip torques of which were controlled using the PD-CTC to minimize the position error. The proposed scheme can be effectively used for neurofeedback training and rehabilitation of lower-limb amputees and paralyzed patients.

Detecting Pilot's Engagement Using fNIRS Connectivity Features in an Automated vs. Manual Landing Scenario

Monitoring pilot's mental states is a relevant approach to mitigate human error and enhance human machine interaction. A promising brain imaging technique to perform such a continuous measure of human mental state under ecological settings is Functional Near-InfraRed Spectroscopy (fNIRS). However, to our knowledge no study has yet assessed the potential of fNIRS connectivity metrics as long as passive Brain Computer Interfaces (BCI) are concerned. Therefore, we designed an experimental scenario in a realistic simulator in which 12 pilots had to perform landings under two contrasted levels of engagement (manual vs. automated). The collected data were used to benchmark the performance of classical oxygenation features (i.e., Average, Peak, Variance, Skewness, Kurtosis, Area Under the Curve, and Slope) and connectivity features (i.e., Covariance, Pearson's, and Spearman's Correlation, Spectral Coherence, and Wavelet Coherence) to discriminate these two landing conditions. Classification performance was obtained by using a shrinkage Linear Discriminant Analysis (sLDA) and a stratified cross validation using each feature alone or by combining them. Our findings disclosed that the connectivity features performed significantly better than the classical concentration metrics with a higher accuracy for the wavelet coherence (average: 65.3/59.9 %, min: 45.3/45.0, max: 80.5/74.7 computed for HbO/HbR signals respectively). A maximum classification performance was obtained by combining the area under the curve with the wavelet coherence (average: 66.9/61.6 %, min: 57.3/44.8, max: 80.0/81.3 computed for HbO/HbR signals respectively). In a general manner all connectivity measures allowed an efficient classification when computed over HbO signals. Those promising results provide methodological cues for further implementation of fNIRS-based passive BCIs.

Enhancing Classification Performance of Functional Near-Infrared Spectroscopy- Brain-Computer Interface Using Adaptive Estimation of General Linear Model Coefficients

In this paper, a novel methodology for enhanced classification of functional near-infrared spectroscopy (fNIRS) signals utilizable in a two-class [motor imagery (MI) and rest; mental rotation (MR) and rest] brain–computer interface (BCI) is presented. First, fNIRS signals corresponding to MI and MR are acquired from the motor and prefrontal cortex, respectively, afterward, filtered to remove physiological noises. Then, the signals are modeled using the general linear model, the coefficients of which are adaptively estimated using the least squares technique. Subsequently, multiple feature combinations of estimated coefficients were used for classification. The best classification accuracies achieved for five subjects, for MI versus rest are 79.5, 83.7, 82.6, 81.4, and 84.1% whereas those for MR versus rest are 85.5, 85.2, 87.8, 83.7, and 84.8%, respectively, using support vector machine. These results are compared with the best classification accuracies obtained using the conventional hemodynamic response. By means of the proposed methodology, the average classification accuracy obtained was significantly higher (p < 0.05). These results serve to demonstrate the feasibility of developing a high-classification-performance fNIRS-BCI.

Application of a common spatial pattern-based algorithm for an fNIRS-based motor imagery brain-computer interface

Motor imagery is one of the most investigated paradigms in the field of brain-computer interfaces (BCIs). The present study explored the feasibility of applying a common spatial pattern (CSP)-based algorithm for a functional near-infrared spectroscopy (fNIRS)-based motor imagery BCI. Ten participants performed kinesthetic imagery of their left- and right-hand movements while 20-channel fNIRS signals were recorded over the motor cortex. The CSP method was implemented to obtain the spatial filters specific for both imagery tasks. The mean, slope, and variance of the CSP filtered signals were taken as features for BCI classification. Results showed that the CSP-based algorithm outperformed two representative channel-wise methods for classifying the two imagery statuses using either data from all channels or averaged data from imagery responsive channels only (oxygenated hemoglobin: CSP-based: 75.3±13.1%; all-channel: 52.3±5.3%; averaged: 64.8±13.2%; deoxygenated hemoglobin: CSP-based: 72.3±13.0%; all-channel: 48.8±8.2%; averaged: 63.3±13.3%). Furthermore, the effectiveness of the CSP method was also observed for the motor execution data to a lesser extent. A partial correlation analysis revealed significant independent contributions from all three types of features, including the often-ignored variance feature. To our knowledge, this is the first study demonstrating the effectiveness of the CSP method for fNIRS-based motor imagery BCIs.

A Gaussian mixture model based adaptive classifier for fNIRS brain-computer interfaces and its testing via simulation

OBJECTIVE

Functional near infra-red spectroscopy (fNIRS) is a promising brain imaging technology for brain-computer interfaces (BCI). Future clinical uses of fNIRS will likely require operation over long time spans, during which neural activation patterns may change. However, current decoders for fNIRS signals are not designed to handle changing activation patterns. The objective of this study is to test via simulations a new adaptive decoder for fNIRS signals, the Gaussian mixture model adaptive classifier (GMMAC).

APPROACH

GMMAC can simultaneously classify and track activation pattern changes without the need for ground-truth labels. This adaptive classifier uses computationally efficient variational Bayesian inference to label new data points and update mixture model parameters, using the previous model parameters as priors. We test GMMAC in simulations in which neural activation patterns change over time and compare to static decoders and unsupervised adaptive linear discriminant analysis classifiers.

MAIN RESULTS

Our simulation experiments show GMMAC can accurately decode under time-varying activation patterns: shifts of activation region, expansions of activation region, and combined contractions and shifts of activation region. Furthermore, the experiments show the proposed method can track the changing shape of the activation region. Compared to prior work, GMMAC performed significantly better than the other unsupervised adaptive classifiers on a difficult activation pattern change simulation: 99% versus  <54% in two-choice classification accuracy.

SIGNIFICANCE

We believe GMMAC will be useful for clinical fNIRS-based brain-computer interfaces, including neurofeedback training systems, where operation over long time spans is required.

Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation

Brain-machine interfaces (BMIs) combine methods, approaches, and concepts derived from neurophysiology, computer science, and engineering in an effort to establish real-time bidirectional links between living brains and artificial actuators. Although theoretical propositions and some proof of concept experiments on directly linking the brains with machines date back to the early 1960s, BMI research only took off in earnest at the end of the 1990s, when this approach became intimately linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals of BMIs are 1) to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations to severely disabled patients. Over the past decade, a wide range of BMI applications have emerged, which considerably expanded these original goals. BMI studies have shown neural control over the movements of robotic and virtual actuators that enact both upper and lower limb functions. Furthermore, BMIs have also incorporated ways to deliver sensory feedback, generated from external actuators, back to the brain. BMI research has been at the forefront of many neurophysiological discoveries, including the demonstration that, through continuous use, artificial tools can be assimilated by the primate brain's body schema. Work on BMIs has also led to the introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term continuous BMI use has been recently implicated with the induction of partial neurological recovery in spinal cord injury patients.

A Non-parametric Approach to the Overall Estimate of Cognitive Load Using NIRS Time Series

We present a non-parametric approach to prediction of the n-back n ∈ {1, 2} task as a proxy measure of mental workload using Near Infrared Spectroscopy (NIRS) data. In particular, we focus on measuring the mental workload through hemodynamic responses in the brain induced by these tasks, thereby realizing the potential that they can offer for their detection in real world scenarios (e.g., difficulty of a conversation). Our approach takes advantage of intrinsic linearity that is inherent in the components of the NIRS time series to adopt a one-step regression strategy. We demonstrate the correctness of our approach through its mathematical analysis. Furthermore, we study the performance of our model in an inter-subject setting in contrast with state-of-the-art techniques in the literature to show a significant improvement on prediction of these tasks (82.50 and 86.40% for female and male participants, respectively). Moreover, our empirical analysis suggest a gender difference effect on the performance of the classifiers (with male data exhibiting a higher non-linearity) along with the left-lateralized activation in both genders with higher specificity in females.

Challenges of implementing a personalized mental task near-infrared spectroscopy brain-computer interface for a non-verbal young adult with motor impairments

PURPOSE

Near-infrared spectroscopy brain-computer interfaces (NIRS-BCIs) have been proposed as potential motor-free communication pathways. This paper documents the challenges of implementing an NIRS-BCI with a non-verbal, severely and congenitally impaired, but cognitively intact young adult.

METHODS

A 5-session personalized mental task NIRS-BCI training paradigm was invoked, whereby participant-specific mental tasks were selected either by the researcher or by the user, on the basis of prior performance or user preference.

RESULTS

Although the personalized mental task selection and training framework had been previously demonstrated with able-bodied participants, the participant was not able to exceed chance-level accuracies. Challenges to the acquisition of BCI control may have included disinclination to BCI training, structural or functional brain atypicalities, heightened emotional arousal and confounding haemodynamic patterns associated with novelty and reward processing.

CONCLUSIONS

Overall, we stress the necessity for further clinical NIRS-BCI research involving non-verbal individuals with severe motor impairments.

Test-retest reliability of the prefrontal response to affective pictures based on functional near-infrared spectroscopy

Functional near-infrared spectroscopy (fNIRS) is being increasingly applied to affective and social neuroscience research; however, the reliability of this method is still unclear. This study aimed to evaluate the test–retest reliability of the fNIRS-based prefrontal response to emotional stimuli. Twenty-six participants viewed unpleasant and neutral pictures, and were simultaneously scanned by fNIRS in two sessions three weeks apart. The reproducibility of the prefrontal activation map was evaluated at three spatial scales (mapwise, clusterwise, and channelwise) at both the group and individual levels. The influence of the time interval was also explored and comparisons were made between longer (intersession) and shorter (intrasession) time intervals. The reliabilities of the activation map at the group level for the mapwise (up to 0.88, the highest value appeared in the intersession assessment) and clusterwise scales (up to 0.91, the highest appeared in the intrasession assessment) were acceptable, indicating that fNIRS may be a reliable tool for emotion studies, especially for a group analysis and under larger spatial scales. However, it should be noted that the individual-level and the channelwise fNIRS prefrontal responses were not sufficiently stable. Future studies should investigate which factors influence reliability, as well as the validity of fNIRS used in emotion studies.

Analysis of Different Classification Techniques for Two-Class Functional Near-Infrared Spectroscopy-Based Brain-Computer Interface

We analyse and compare the classification accuracies of six different classifiers for a two-class mental task (mental arithmetic and rest) using functional near-infrared spectroscopy (fNIRS) signals. The signals of the mental arithmetic and rest tasks from the prefrontal cortex region of the brain for seven healthy subjects were acquired using a multichannel continuous-wave imaging system. After removal of the physiological noises, six features were extracted from the oxygenated hemoglobin (HbO) signals. Two- and three-dimensional combinations of those features were used for classification of mental tasks. In the classification, six different modalities, linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), k-nearest neighbour (kNN), the Naïve Bayes approach, support vector machine (SVM), and artificial neural networks (ANN), were utilized. With these classifiers, the average classification accuracies among the seven subjects for the 2- and 3-dimensional combinations of features were 71.6, 90.0, 69.7, 89.8, 89.5, and 91.4% and 79.6, 95.2, 64.5, 94.8, 95.2, and 96.3%, respectively. ANN showed the maximum classification accuracies: 91.4 and 96.3%. In order to validate the results, a statistical significance test was performed, which confirmed that the p values were statistically significant relative to all of the other classifiers (p < 0.005) using HbO signals.

Toward more intuitive brain-computer interfacing: classification of binary covert intentions using functional near-infrared spectroscopy

In traditional brain-computer interface (BCI) studies, binary communication systems have generally been implemented using two mental tasks arbitrarily assigned to “yes” or “no” intentions (e.g., mental arithmetic calculation for “yes”). A recent pilot study performed with one paralyzed patient showed the possibility of a more intuitive paradigm for binary BCI communications, in which the patient’s internal yes/no intentions were directly decoded from functional near-infrared spectroscopy (fNIRS). We investigated whether such an “fNIRS-based direct intention decoding” paradigm can be reliably used for practical BCI communications. Eight healthy subjects participated in this study, and each participant was administered 70 disjunctive questions. Brain hemodynamic responses were recorded using a multichannel fNIRS device, while the participants were internally expressing “yes” or “no” intentions to each question. Different feature types, feature numbers, and time window sizes were tested to investigate optimal conditions for classifying the internal binary intentions. About 75% of the answers were correctly classified when the individual best feature set was employed (75.89% ± 1.39 and 74.08% ± 2.87 for oxygenated and deoxygenated hemoglobin responses, respectively), which was significantly higher than a random chance level (68.57% for p < 0.001). The kurtosis feature showed the highest mean classification accuracy among all feature types. The grand-averaged hemodynamic responses showed that wide brain regions are associated with the processing of binary implicit intentions. Our experimental results demonstrated that direct decoding of internal binary intention has the potential to be used for implementing more intuitive and user-friendly communication systems for patients with motor disabilities.

Single trial classification of fNIRS-based brain-computer interface mental arithmetic data: a comparison between different classifiers

Functional near infrared spectroscopy (fNIRS) is an emerging technique for the in-vivo assessment of functional activity of the cerebral cortex as well as in the field of brain-computer-interface (BCI) research. A common challenge for the utilization of fNIRS for BCIs is a stable and reliable single trial classification of the recorded spatio-temporal hemodynamic patterns. Many different classification methods are available, but up to now, not more than two different classifiers were evaluated and compared on one data set. In this work, we overcome this issue by comparing five different classification methods on mental arithmetic fNIRS data: linear discriminant analysis (LDA), quadratic discriminant analysis (QDA), support vector machines (SVM), analytic shrinkage regularized LDA (sLDA), and analytic shrinkage regularized QDA (sQDA). Depending on the used method and feature type (oxy-Hb or deoxy-Hb), achieved classification results vary between 56.1 % (deoxy-Hb/QDA) and 86.6% (oxy-Hb/SVM). We demonstrated that regularized classifiers perform significantly better than non-regularized ones. Considering simplicity and computational effort, we recommend the use of sLDA for fNIRS-based BCIs.

Determining Optimal Feature-Combination for LDA Classification of Functional Near-Infrared Spectroscopy Signals in Brain-Computer Interface Application

In this study, we determine the optimal feature-combination for classification of functional near-infrared spectroscopy (fNIRS) signals with the best accuracies for development of a two-class brain-computer interface (BCI). Using a multi-channel continuous-wave imaging system, mental arithmetic signals are acquired from the prefrontal cortex of seven healthy subjects. After removing physiological noises, six oxygenated and deoxygenated hemoglobin (HbO and HbR) features-mean, slope, variance, peak, skewness and kurtosis-are calculated. All possible 2- and 3-feature combinations of the calculated features are then used to classify mental arithmetic vs. rest using linear discriminant analysis (LDA). It is found that the combinations containing mean and peak values yielded significantly higher (p < 0.05) classification accuracies for both HbO and HbR than did all of the other combinations, across all of the subjects. These results demonstrate the feasibility of achieving high classification accuracies using mean and peak values of HbO and HbR as features for classification of mental arithmetic vs. rest for a two-class BCI.

Decoding four different sound-categories in the auditory cortex using functional near-infrared spectroscopy

The ability of the auditory cortex in the brain to distinguish different sounds is important in daily life. This study investigated whether activations in the auditory cortex caused by different sounds can be distinguished using functional near-infrared spectroscopy (fNIRS). The hemodynamic responses (HRs) in both hemispheres using fNIRS were measured in 18 subjects while exposing them to four sound categories (English-speech, non-English-speech, annoying sounds, and nature sounds). As features for classifying the different signals, the mean, slope, and skewness of the oxy-hemoglobin (HbO) signal were used. With regard to the language-related stimuli, the HRs evoked by understandable speech (English) were observed in a broader brain region than were those evoked by non-English speech. Also, the magnitudes of the HbO signals evoked by English-speech were higher than those of non-English speech. The ratio of the peak values of non-English and English speech was 72.5%. Also, the brain region evoked by annoying sounds was wider than that by nature sounds. However, the signal strength for nature sounds was stronger than that for annoying sounds. Finally, for brain-computer interface (BCI) purposes, the linear discriminant analysis (LDA) and support vector machine (SVM) classifiers were applied to the four sound categories. The overall classification performance for the left hemisphere was higher than that for the right hemisphere. Therefore, for decoding of auditory commands, the left hemisphere is recommended. Also, in two-class classification, the annoying vs. nature sounds comparison provides a higher classification accuracy than the English vs. non-English speech comparison. Finally, LDA performs better than SVM.