Mental Task Evaluation for Hybrid NIRS-EEG Brain-Computer Interfaces

Assessing Consciousness in Patients With Disorders of Consciousness Using a Musical Stimulation Paradigm and Verifiable Criteria

Numerous studies have shown that musical stimulation can activate corresponding functional brain areas. Electroencephalogram (EEG) activity during musical stimulation can be used to assess the consciousness states of patients with disorders of consciousness (DOC). In this study, a musical stimulation paradigm and verifiable criteria were used for consciousness assessment. Twenty-nine participants (13 healthy subjects, 6 patients in a minimally conscious state (MCS) and 10 patients in a vegetative state (VS)) were recruited, and EEG signals were collected while participants listened to preferred and relaxing music. Fusion features based on differential entropy (DE), common spatial pattern (CSP), and EEG-based network pattern (ENP) features were extracted from EEG signals, and a convolutional neural network-long short-term memory (CNN-LSTM) model was employed to classify preferred and relaxing music.The results showed that the average classification accuracy for healthy subjects reached 85.58%. For two of the patients in the MCS group, the classification accuracies reached 78.18% and 66.14%, and they were diagnosed with emergence from MCS (EMCS) two months later. The accuracies of three patients in the VS group were 58.18%, 64.32% and 62.05%, with two patients showing slight increases in scale scores. Our study suggests that musical stimulation could be an effective method for consciousness detection, with significant diagnostic implications for patients with DOC.

Mental workload assessment by monitoring brain, heart, and eye with six biomedical modalities during six cognitive tasks

Introduction

The efficiency and safety of complex high precision human-machine systems such as in aerospace and robotic surgery are closely related to the cognitive readiness, ability to manage workload, and situational awareness of their operators. Accurate assessment of mental workload could help in preventing operator error and allow for pertinent intervention by predicting performance declines that can arise from either work overload or under stimulation. Neuroergonomic approaches based on measures of human body and brain activity collectively can provide sensitive and reliable assessment of human mental workload in complex training and work environments.

Methods

In this study, we developed a new six-cognitive-domain task protocol, coupling it with six biomedical monitoring modalities to concurrently capture performance and cognitive workload correlates across a longitudinal multi-day investigation. Utilizing two distinct modalities for each aspect of cardiac activity (ECG and PPG), ocular activity (EOG and eye-tracking), and brain activity (EEG and fNIRS), 23 participants engaged in four sessions over 4 weeks, performing tasks associated with working memory, vigilance, risk assessment, shifting attention, situation awareness, and inhibitory control.

Results

The results revealed varying levels of sensitivity to workload within each modality. While certain measures exhibited consistency across tasks, neuroimaging modalities, in particular, unveiled meaningful differences between task conditions and cognitive domains.

Discussion

This is the first comprehensive comparison of these six brain-body measures across multiple days and cognitive domains. The findings underscore the potential of wearable brain and body sensing methods for evaluating mental workload. Such comprehensive neuroergonomic assessment can inform development of next generation neuroadaptive interfaces and training approaches for more efficient human-machine interaction and operator skill acquisition.

Towards user-centric BCI design: Markov chain-based user assessment for mental imagery EEG-BCIs

Objective.While electroencephalography (EEG)-based brain-computer interfaces (BCIs) have many potential clinical applications, their use is impeded by poor performance for many users. To improve BCI performance, either via enhanced signal processing or user training, it is critical to understand and describe each user's ability to perform mental control tasks and produce discernible EEG patterns. While classification accuracy has predominantly been used to assess user performance, limitations and criticisms of this approach have emerged, thus prompting the need to develop novel user assessment approaches with greater descriptive capability. Here, we propose a combination of unsupervised clustering and Markov chain models to assess and describe user skill.Approach.Using unsupervisedK-means clustering, we segmented the EEG signal space into regions representing pattern states that users could produce. A user's movement through these pattern states while performing different tasks was modeled using Markov chains. Finally, using the steady-state distributions and entropy rates of the Markov chains, we proposed two metricstaskDistinctandrelativeTaskInconsistencyto assess, respectively, a user's ability to (i) produce distinct task-specific patterns for each mental task and (ii) maintain consistent patterns during individual tasks.Main results.Analysis of data from 14 adolescents using a three-class BCI revealed significant correlations between thetaskDistinctandrelativeTaskInconsistencymetrics and classification F1 score. Moreover, analysis of the pattern states and Markov chain models yielded descriptive information regarding user performance not immediately apparent from classification accuracy.Significance.Our proposed user assessment method can be used in concert with classifier-based analysis to further understand the extent to which users produce task-specific, time-evolving EEG patterns. In turn, this information could be used to enhance user training or classifier design.

EEG Amplitude Modulation Analysis across Mental Tasks: Towards Improved Active BCIs

Brain-computer interface (BCI) technology has emerged as an influential communication tool with extensive applications across numerous fields, including entertainment, marketing, mental state monitoring, and particularly medical neurorehabilitation. Despite its immense potential, the reliability of BCI systems is challenged by the intricacies of data collection, environmental factors, and noisy interferences, making the interpretation of high-dimensional electroencephalogram (EEG) data a pressing issue. While the current trends in research have leant towards improving classification using deep learning-based models, our study proposes the use of new features based on EEG amplitude modulation (AM) dynamics. Experiments on an active BCI dataset comprised seven mental tasks to show the importance of the proposed features, as well as their complementarity to conventional power spectral features. Through combining the seven mental tasks, 21 binary classification tests were explored. In 17 of these 21 tests, the addition of the proposed features significantly improved classifier performance relative to using power spectral density (PSD) features only. Specifically, the average kappa score for these classifications increased from 0.57 to 0.62 using the combined feature set. An examination of the top-selected features showed the predominance of the AM-based measures, comprising over 77% of the top-ranked features. We conclude this paper with an in-depth analysis of these top-ranked features and discuss their potential for use in neurophysiology.

Multimodal resting-state connectivity predicts affective neurofeedback performance

Neurofeedback has been suggested as a potential complementary therapy to different psychiatric disorders. Of interest for this approach is the prediction of individual performance and outcomes. In this study, we applied functional connectivity-based modeling using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) modalities to (i) investigate whether resting-state connectivity predicts performance during an affective neurofeedback task and (ii) evaluate the extent to which predictive connectivity profiles are correlated across EEG and fNIRS techniques. The fNIRS oxyhemoglobin and deoxyhemoglobin concentrations and the EEG beta and gamma bands modulated by the alpha frequency band (beta-m-alpha and gamma-m-alpha, respectively) recorded over the frontal cortex of healthy subjects were used to estimate functional connectivity from each neuroimaging modality. For each connectivity matrix, relevant edges were selected in a leave-one-subject-out procedure, summed into "connectivity summary scores" (CSS), and submitted as inputs to a support vector regressor (SVR). Then, the performance of the left-out-subject was predicted using the trained SVR model. Linear relationships between the CSS across both modalities were evaluated using Pearson's correlation. The predictive model showed a mean absolute error smaller than 20%, and the fNIRS oxyhemoglobin CSS was significantly correlated with the EEG gamma-m-alpha CSS (r = -0.456, p = 0.030). These results support that pre-task electrophysiological and hemodynamic resting-state connectivity are potential predictors of neurofeedback performance and are meaningfully coupled. This investigation motivates the use of joint EEG-fNIRS connectivity as outcome predictors, as well as a tool for functional connectivity coupling investigation.

Real-Time Algorithm for Detrended Cross-Correlation Analysis of Long-Range Coupled Processes

Assessing power-law cross-correlations between a pair - or among a set - of processes is of great significance in diverse fields of analyses ranging from neuroscience to financial markets. In most cases such analyses are computationally expensive and thus carried out offline once the entire signal is obtained. However, many applications - such as mental state monitoring or financial forecasting - call for fast algorithms capable of estimating scale-free coupling in real time. Detrended cross-correlation analysis (DCCA), a generalization of the detrended fluctuation analysis (DFA) to the bivariate domain, has been introduced as a method designed to quantify power-law cross-correlations between a pair of non-stationary signals. Later, in analogy with the Pearson cross-correlation coefficient, DCCA was adapted to the detrended cross-correlation coefficient (DCCC), however as of now no online algorithms were provided for either of these analysis techniques. Here we introduce a new formula for obtaining the scaling functions in real time for DCCA. Moreover, the formula can be generalized via matrix notation to obtain the scaling relationship between not only a pair of signals, but also all possible pairs among a set of signals at the same time. This includes parallel estimation of the DFA scaling function of each individual process as well, thus allowing also for real-time acquisition of DCCC. The proposed algorithm matches its offline variants in precision, while being substantially more efficient in terms of execution time. We demonstrate that the method can be utilized for mental state monitoring on multi-channel electroencephalographic recordings obtained in eyes-closed and eyes-open resting conditions.

Multi-Modal Integration of EEG-fNIRS for Characterization of Brain Activity Evoked by Preferred Music

Music can effectively improve people's emotions, and has now become an effective auxiliary treatment method in modern medicine. With the rapid development of neuroimaging, the relationship between music and brain function has attracted much attention. In this study, we proposed an integrated framework of multi-modal electroencephalogram (EEG) and functional near infrared spectroscopy (fNIRS) from data collection to data analysis to explore the effects of music (especially personal preferred music) on brain activity. During the experiment, each subject was listening to two different kinds of music, namely personal preferred music and neutral music. In analyzing the synchronization signals of EEG and fNIRS, we found that music promotes the activity of the brain (especially the prefrontal lobe), and the activation induced by preferred music is stronger than that of neutral music. For the multi-modal features of EEG and fNIRS, we proposed an improved Normalized-ReliefF method to fuse and optimize them and found that it can effectively improve the accuracy of distinguishing between the brain activity evoked by preferred music and neutral music (up to 98.38%). Our work provides an objective reference based on neuroimaging for the research and application of personalized music therapy.

Data Processing in Functional Near-Infrared Spectroscopy (fNIRS) Motor Control Research

FNIRS pre-processing and processing methodologies are very important-how a researcher chooses to process their data can change the outcome of an experiment. The purpose of this review is to provide a guide on fNIRS pre-processing and processing techniques pertinent to the field of human motor control research. One hundred and twenty-three articles were selected from the motor control field and were examined on the basis of their fNIRS pre-processing and processing methodologies. Information was gathered about the most frequently used techniques in the field, which included frequency cutoff filters, wavelet filters, smoothing filters, and the general linear model (GLM). We discuss the methodologies of and considerations for these frequently used techniques, as well as those for some alternative techniques. Additionally, general considerations for processing are discussed.

Subject-Specific feature selection for near infrared spectroscopy based brain-computer interfaces

BACKGROUND AND OBJECTIVE

Brain-computer interfaces (BCIs) enable people to control an external device by analyzing the brain's neural activity. Functional near-infrared spectroscopy (fNIRS), which is an emerging optical imaging technique, is frequently used in non-invasive BCIs. Determining the subject-specific features is an important concern in enhancing the classification accuracy as well as reducing the complexity of fNIRS based BCI systems. In this study, the effectiveness of subject-specific feature selection on classification accuracy of fNIRS signals is examined.

METHODS

In order to determine the subject-specific optimal feature subsets, stepwise regression analysis based on sequential feature selection (SWR-SFS) and ReliefF methods were employed. Feature selection is applied on time-domain features of fNIRS signals such as mean, slope, peak, skewness and kurtosis values of signals. Linear discriminant analysis, k nearest neighborhood and support vector machines are employed to evaluate the performance of the selected feature subsets. The proposed techniques are validated on benchmark motor imagery (MI) and mental arithmetic (MA) based fNIRS datasets collected from 29 healthy subjects.

RESULTS

Both SWR-SFS and reliefF feature selection methods have significantly improved the classification accuracy. However, the best results (88.67% (HbR) and 86.43% (HbO) for MA dataset and 77.01% (HbR) and 71.32% (HbO) for MI dataset) were achieved using SWR-SFS while feature selection provided extremely high feature reduction rates (89.50% (HbR) and 93.99% (HbO) for MA dataset and 94.04% (HbR) and 97.73% (HbO) for MI dataset).

CONCLUSIONS

The results of the study indicate that employing feature selection improves both MA and MI-based fNIRS signals classification performance significantly.

Functional Near-Infrared Spectroscopy Indicates That Asymmetric Right Hemispheric Activation in Mental Rotation of a Jigsaw Puzzle Decreases With Task Difficulty

Mental rotation (MR) is a cognitive skill whose neural dynamics are still a matter of debate as previous neuroimaging studies have produced controversial results. In order to investigate the underlying neurophysiology of MR, hemodynamic responses from the prefrontal cortex of 14 healthy subjects were recorded with functional near-infrared spectroscopy (fNIRS) during a novel MR task that had three categorical difficulty levels. Hemodynamic activity strength (HAS) parameter, which reflects the ratio of brain activation during the task to the baseline activation level, was used to assess the prefrontal cortex activation localization and strength. Behavioral data indicated that the MR requiring conditions are more difficult than the condition that did not require MR. The right dorsolateral prefrontal cortex (DLPFC) was found to be active in all conditions and to be the dominant region in the easiest task while more complex tasks showed widespread bilateral prefrontal activation. A significant increase in left DLPFC activation was observed with increasing task difficulty. Significantly higher right DLPFC activation was observed when the incongruent trials were contrasted against the congruent trials, which implied the possibility of a robust error or conflict-monitoring process during the incongruent trials. Our results showed that the right DLPFC is a core region for the processing of MR tasks regardless of the task complexity and that the left DLPFC is involved to a greater extent with increasing task complexity, which is consistent with the previous neuroimaging literature.

Proposing a convolutional neural network for stress assessment by means of derived heart rate from functional near infrared spectroscopy

BACKGROUND

Stress is known as one of the major factors threatening human health. A large number of studies have been performed in order to either assess or relieve stress by analyzing the brain and heart-related signals.

METHOD

In this study, a method based on the Convolutional Neural Network (CNN) approach is proposed to assess stress induced by the Montreal Imaging Stress Task. The proposed model is trained on the heart rate signal derived from functional Near-Infrared Spectroscopy (fNIRS), which is referred to as HRF. In this regard, fNIRS signals of 20 healthy volunteers were recorded using a configuration of 23 channels located on the prefrontal cortex. The proposed deep learning system consists of two main parts where in the first part, the one-dimensional convolutional neural network is employed to build informative activation maps, and then in the second part, a stack of deep fully connected layers is used to predict the stress existence probability. Thereafter, the employed CNN method is compared with the Dense Neural Network, Support Vector Machine, and Random Forest regarding various classification metrics.

RESULTS

Results clearly showed the superiority of CNN over all other methods. Additionally, the trained HRF model significantly outperforms the model trained on the filtered fNIRS signals, where the HRF model could achieve 98.69 ± 0.45% accuracy, which is 10.09% greater than the accuracy obtained by the fNIRS model.

CONCLUSIONS

Employment of the proposed deep learning system trained on the HRF measurements leads to higher stress classification accuracy than the accuracy reported in the existing studies where the same experimental procedure has been done. Besides, the proposed method suggests better stability with lower variation in prediction. Furthermore, its low computational cost opens up the possibility to be applied in real-time monitoring of stress assessment.

Resting-State NIRS-EEG in Unresponsive Patients with Acute Brain Injury: A Proof-of-Concept Study

BACKGROUND

Neurovascular-based imaging techniques such as functional MRI (fMRI) may reveal signs of consciousness in clinically unresponsive patients but are often subject to logistical challenges in the intensive care unit (ICU). Near-infrared spectroscopy (NIRS) is another neurovascular imaging technique but low cost, can be performed serially at the bedside, and may be combined with electroencephalography (EEG), which are important advantages compared to fMRI. Combined NIRS-EEG, however, has never been evaluated for the assessment of neurovascular coupling and consciousness in acute brain injury.

METHODS

We explored resting-state oscillations in eight-channel NIRS oxyhemoglobin and eight-channel EEG band-power signals to assess neurovascular coupling, the prerequisite for neurovascular-based imaging detection of consciousness, in patients with acute brain injury in the ICU (n = 9). Conscious neurological patients from step-down units and wards served as controls (n = 14). Unsupervised adaptive mixture-independent component analysis (AMICA) was used to correlate NIRS-EEG data with levels of consciousness and clinical outcome.

RESULTS

Neurovascular coupling between NIRS oxyhemoglobin (0.07-0.13 Hz) and EEG band-power (1-12 Hz) signals at frontal areas was sensitive and prognostic to changing consciousness levels. AMICA revealed a mixture of five models from EEG data, with the relative probabilities of these models reflecting levels of consciousness over multiple days, although the accuracy was less than 85%. However, when combined with two channels of bilateral frontal neurovascular coupling, weighted k-nearest neighbor classification of AMICA probabilities distinguished unresponsive patients from conscious controls with > 90% accuracy (positive predictive value 93%, false discovery rate 7%) and, additionally, identified patients who subsequently failed to recover consciousness with > 99% accuracy.

DISCUSSION

We suggest that NIRS-EEG for monitoring of acute brain injury in the ICU is worthy of further exploration. Normalization of neurovascular coupling may herald recovery of consciousness after acute brain injury.

Elevation of intracranial pressure affects the relationship between hemoglobin concentration and neuronal activation in human somatosensory cortex

During neuronal activation, a local decrease of deoxygenated hemoglobin concentration (deoxy‐Hb) occurs which is the basis of functional brain imaging with blood oxygenation level dependent functional magnetic resonance imaging (BOLD‐fMRI). Elevated intracranial pressure (eICP) has been shown to impair functional deoxy‐Hb changes. This study investigated this effect and its relation to the underlying neuronal activity in the human primary somatosensory cortex (SI). Functional near‐infrared spectroscopy (fNIRS) during somatosensory evoked potentials (SEP) monitoring was performed on 75 subjects during conditions of median nerve stimulation (MNS) and resting state, combined with normal breathing (NB) and eICP by escalating breathing maneuvers (breath holding [BH], Valsalva maneuver with 15 mmHg [V15] and 35 mmHg expiratory pressure [V35]). During NB, fNIRS revealed a typical oxygenated hemoglobin concentration (oxy‐Hb) increase with deoxy‐Hb decrease during MNS enabling SI brain mapping. Breathing maneuvers associated eICP produced a known global change of oxy‐Hb and deoxy‐Hb with and without MNS. When subtracting measurements during resting state from measurements during MNS, neither functional oxy‐Hb nor deoxy‐Hb changes could be recovered while SEPs remained unchanged. In conclusion, Valsalva‐induced eICP prevents oxy‐Hb and deoxy‐Hb changes during neuronal activation in SI. This finding raises questions on the validity of oxy‐Hb‐ and deoxy‐Hb‐based brain imaging (e.g., BOLD‐fMRI) during eICP.

A Ternary Hybrid EEG-NIRS Brain-Computer Interface for the Classification of Brain Activation Patterns during Mental Arithmetic, Motor Imagery, and Idle State

The performance of a brain-computer interface (BCI) can be enhanced by simultaneously using two or more modalities to record brain activity, which is generally referred to as a hybrid BCI. To date, many BCI researchers have tried to implement a hybrid BCI system by combining electroencephalography (EEG) and functional near-infrared spectroscopy (NIRS) to improve the overall accuracy of binary classification. However, since hybrid EEG-NIRS BCI, which will be denoted by hBCI in this paper, has not been applied to ternary classification problems, paradigms and classification strategies appropriate for ternary classification using hBCI are not well investigated. Here we propose the use of an hBCI for the classification of three brain activation patterns elicited by mental arithmetic, motor imagery, and idle state, with the aim to elevate the information transfer rate (ITR) of hBCI by increasing the number of classes while minimizing the loss of accuracy. EEG electrodes were placed over the prefrontal cortex and the central cortex, and NIRS optodes were placed only on the forehead. The ternary classification problem was decomposed into three binary classification problems using the "one-versus-one" (OVO) classification strategy to apply the filter-bank common spatial patterns filter to EEG data. A 10 × 10-fold cross validation was performed using shrinkage linear discriminant analysis (sLDA) to evaluate the average classification accuracies for EEG-BCI, NIRS-BCI, and hBCI when the meta-classification method was adopted to enhance classification accuracy. The ternary classification accuracies for EEG-BCI, NIRS-BCI, and hBCI were 76.1 ± 12.8, 64.1 ± 9.7, and 82.2 ± 10.2%, respectively. The classification accuracy of the proposed hBCI was thus significantly higher than those of the other BCIs (p < 0.005). The average ITR for the proposed hBCI was calculated to be 4.70 ± 1.92 bits/minute, which was 34.3% higher than that reported for a previous binary hBCI study.