Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks

The detection of the cognitive tasks performed by a subject during data acquisition of a neuroimaging method has a wide range of applications: functioning of brain-computer interface (BCI), detection of neuronal disorders, neurorehabilitation for disabled patients, and many others. Recent studies show that the combination or fusion of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) demonstrates improved classification and detection performance compared to sole-EEG and sole-fNIRS. Deep learning (DL) networks are suitable for the classification of large volume time-series data like EEG and fNIRS. This study performs the decision fusion of EEG and fNIRS. The classification of EEG, fNIRS, and decision-fused EEG-fNIRSinto cognitive task labels is performed by DL networks. Two different open-source datasets of simultaneously recorded EEG and fNIRS are examined in this study. Dataset 01 is comprised of 26 subjects performing 3 cognitive tasks: n-back, discrimination or selection response (DSR), and word generation (WG). After data acquisition, fNIRS is converted to oxygenated hemoglobin (HbO2) and deoxygenated hemoglobin (HbR) in Dataset 01. Dataset 02 is comprised of 29 subjects who performed 2 tasks: motor imagery and mental arithmetic. The classification procedure of EEG and fNIRS (or HbO2, HbR) are carried out by 7 DL classifiers: convolutional neural network (CNN), long short-term memory network (LSTM), gated recurrent unit (GRU), CNN-LSTM, CNN-GRU, LSTM-GRU, and CNN-LSTM-GRU. After the classification of single modalities, their prediction scores or decisions are combined to obtain the decision-fused modality. The classification performance is measured by overall accuracy and area under the ROC curve (AUC). The highest accuracy and AUC recorded in Dataset 01 are 96% and 100% respectively; both by the decision fusion modality using CNN-LSTM-GRU. For Dataset 02, the highest accuracy and AUC are 82.76% and 90.44% respectively; both by the decision fusion modality using CNN-LSTM. The experimental result shows that decision-fused EEG-HbO2-HbR and EEG-fNIRSdeliver higher performances compared to their constituent unimodalities in most cases. For DL classifiers, CNN-LSTM-GRU in Dataset 01 and CNN-LSTM in Dataset 02 yield the highest performance.

On decoding of rapid motor imagery in a diverse population using a high-density NIRS device

Introduction

Functional near-infrared spectroscopy (fNIRS) aims to infer cognitive states such as the type of movement imagined by a study participant in a given trial using an optical method that can differentiate between oxygenation states of blood in the brain and thereby indirectly between neuronal activity levels. We present findings from an fNIRS study that aimed to test the applicability of a high-density (>3000 channels) NIRS device for use in short-duration (2 s) left/right hand motor imagery decoding in a diverse, but not explicitly balanced, subject population. A side aim was to assess relationships between data quality, self-reported demographic characteristics, and brain-computer interface (BCI) performance, with no subjects rejected from recruitment or analysis.

Methods

BCI performance was quantified using several published methods, including subject-specific and subject-independent approaches, along with a high-density fNIRS decoder previously validated in a separate study.

Results

We found that decoding of motor imagery on this population proved extremely challenging across all tested methods. Overall accuracy of the best-performing method (the high-density decoder) was 59.1 +/- 6.7% after excluding subjects where almost no optode-scalp contact was made over motor cortex and 54.7 +/- 7.6% when all recorded sessions were included. Deeper investigation revealed that signal quality, hemodynamic responses, and BCI performance were all strongly impacted by the hair phenotypical and demographic factors under investigation, with over half of variance in signal quality explained by demographic factors alone.

Discussion

Our results contribute to the literature reporting on challenges in using current-generation NIRS devices on subjects with long, dense, dark, and less pliable hair types along with the resulting potential for bias. Our findings confirm the need for increased focus on these populations, accurate reporting of data rejection choices across subject intake, curation, and final analysis in general, and signal a need for NIRS optode designs better optimized for the general population to facilitate more robust and inclusive research outcomes.

Effective DBS treatment improves neural information transmission of patients with disorders of consciousness: an fNIRS study

Objective.Deep brain stimulation (DBS) is a potential treatment that promotes the recovery of patients with disorders of consciousness (DOC). This study quantified the changes in consciousness and the neuromodulation effect of DBS on patients with DOC.Approach.Eleven patients were recruited for this study which consists of three conditions: 'Pre' (two days before DBS surgery), 'Post-On' (one month after surgery with stimulation), and 'Post-Off' (one month after surgery without stimulation). Functional near-infrared spectroscopy (fNIRS) was recorded from the frontal lobe, parietal lobe, and occipital lobe of patients during the experiment of auditory stimuli paradigm, in parallel with the coma recovery scale-revised (CRS-R) assessment. The brain hemodynamic states were defined and state transition acceleration was taken to quantify the information transmission strength of the brain network. Linear regression analysis was conducted between the changes in regional and global indicators and the changes in the CRS-R index.Main results.Significant correlation was observed between the changes in the global transition acceleration indicator and the changes in the CRS-R index (slope = 55.910,p< 0.001,R2= 0.732). For the regional indicators, similar correlations were found between the changes in the frontal lobe and parietal lobe indicators and the changes in the CRS-R index (slope = 46.612,p< 0.01,R2= 0.694; slope = 47.491,p< 0.01,R2= 0.676).Significance.Our study suggests that fNIRS-based brain hemodynamics transition analysis can signify the neuromodulation effect of DBS treatment on patients with DOC, and the transition acceleration indicator is a promising brain functional marker for DOC.

Temporo-Parietal cortex activation during motor imagery in older adults: A case study of Baduanjin

Age-associated cognitive and motor decline is related to central nervous system injury in older adults. Motor imagery training (MIT), as an emerging rehabilitative intervention, can activate neural basis similar to that in actual exercise, so as to promote motor function in older adults. The complex motor skills rely on the functional integration of the cerebral cortex. Understanding the neural mechanisms underlying motor imagery in older adults would support its application in motor rehabilitation and slowing cognitive decline. Based on this, the present study used functional near infrared spectroscopy (fNIRS) to record the changes in oxygen saturation in older adults (20 participants; mean age, 64.8 ± 4.5 years) during Baduanjin motor execution (ME) and motor imagery (MI). ME significantly activated the left postcentral gyrus, while the oxy-hemoglobin concentration in the right middle temporal gyrus increased significantly during motor imagery. These results indicate that advanced ME activates brain regions related to sensorimotor function, and MI increases the activation of the frontal-parietal cortex related to vision. In older adults, MI overactivated the temporo-parietal region associated with vision, and tend to be activated in the right brain.

Impacts of simplifying articulation movements imagery to speech imagery BCI performance

Objective.Speech imagery (SI) can be used as a reliable, natural, and user-friendly activation task for the development of brain-computer interface (BCI), which empowers individuals with severe disabilities to interact with their environment. The functional near-infrared spectroscopy (fNIRS) is advanced as one of the most suitable brain imaging methods for developing BCI systems owing to its advantages of being non-invasive, portable, insensitive to motion artifacts, and having relatively high spatial resolution.Approach.To improve the classification performance of SI BCI based on fNIRS, a novel paradigm was developed in this work by simplifying the articulation movements in SI to make the articulation movement differences clearer between different words imagery tasks. A SI BCI was proposed to directly answer questions by covertly rehearsing the word '' or '' ('yes' or 'no' in English), and an unconstrained rest task also was contained in this BCI. The articulation movements of SI were simplified by retaining only the movements of the jaw and lips of vowels in Chinese Pinyin for words '' and ''.Main results.Compared with conventional speech imagery, simplifying the articulation movements in SI could generate more different brain activities among different tasks, which led to more differentiable temporal features and significantly higher classification performance. The average 3-class classification accuracies of the proposed paradigm across all 20 participants reached 69.6% and 60.2% which were about 10.8% and 5.6% significantly higher than those of the conventional SI paradigm operated in the 0-10 s and 0-2.5 s time windows, respectively.Significance.These results suggested that simplifying the articulation movements in SI is promising for improving the classification performance of intuitive BCIs based on speech imagery.

Classification of Textile Samples Using Data Fusion Combining Near- and Mid-Infrared Spectral Information

There is an urgent need to reuse and recycle textile fibers, since today, low recycling rates are achieved. Accurate classification methods for post-consumer textile waste are needed in the short term for a higher circularity in the textile and fashion industries. This paper compares different spectroscopic data from textile samples in order to correctly classify the textile samples. The accurate classification of textile waste results in higher recycling rates and a better quality of the recycled materials. The data fusion of near- and mid-infrared spectra is compared with single-spectrum information. The classification results show that data fusion is a better option, providing more accurate classification results, especially for difficult classification problems where the classes are wide and close to one another. The experimental results presented in this paper prove that the data fusion of near- and mid-infrared spectra is a good option for accurate textile-waste classification, since this approach allows the classification results to be significantly improved.

Involvement of the Rostromedial Prefrontal Cortex in Human-Robot Interaction: fNIRS Evidence From a Robot-Assisted Motor Task

Assistive exoskeleton robots are being widely applied in neurorehabilitation to improve upper-limb motor and somatosensory functions. During robot-assisted exercises, the central nervous system appears to highly attend to external information-processing (IP) to efficiently interact with robotic assistance. However, the neural mechanisms underlying this process remain unclear. The rostromedial prefrontal cortex (rmPFC) may be the core of the executive resource allocation that generates biases in the allocation of processing resources toward an external IP according to current behavioral demands. Here, we used functional near-infrared spectroscopy to investigate the cortical activation associated with executive resource allocation during a robot-assisted motor task. During data acquisition, participants performed a right-arm motor task using elbow flexion-extension movements in three different loading conditions: robotic assistive loading (ROB), resistive loading (RES), and non-loading (NON). Participants were asked to strive for kinematic consistency in their movements. A one-way repeated measures analysis of variance and general linear model-based methods were employed to examine task-related activity. We demonstrated that hemodynamic responses in the ventral and dorsal rmPFC were higher during ROB than during NON. Moreover, greater hemodynamic responses in the ventral rmPFC were observed during ROB than during RES. Increased activation in ventral and dorsal rmPFC subregions may be involved in the executive resource allocation that prioritizes external IP during human-robot interactions. In conclusion, these findings provide novel insights regarding the involvement of executive control during a robot-assisted motor task.

An Energy-Efficient Wearable Functional Near-infrared Spectroscopy System Employing Dual-level Adaptive Sampling Technique

Functional near-infrared spectroscopy (fNIRS) is a powerful medical imaging tool in brain science and psychology, it can also be employed in brain-computer interface (BCI) due to its noninvasive and artifact-less-sensitive characteristics. Conventional ways to detect large-area brain activity using near-infrared (NIR) technology are based on Time-division or Frequency-division modulation technique, which traverses all physical sensory channels in a specific period. To achieve higher imaging resolution or brain-tasks classification accuracy, the NIRS system require higher density and more channels, which conflict with the limited battery capacity. Inspired by the functional atlas of the human brain, this paper proposes a spatial adaptive sampling (SAS) method. It can change the active channel pattern of the fNIRS system to match with the real-time brain activity, to increase the energy efficiency without significant reduction on the brain imaging quality or the accuracy of brain activity classification. Therefore, the number of the averaging enabled channels will be dramatically reduced in practice. To verify the proposed SAS technique, a wearable and flexible NIRS system has been implemented, in which each channel of light-emitting diode (LED) drive circuits and photodiode (PD) detection circuits can be power gated independently. Brain task experiments have been conducted to validate the proposed method, the power consumption of the LED drive module is reduced by 46.58% compared to that without SAS technology while maintaining an average brain imaging PSNR (Peak Signal to Noise Ratio) of 35 dB. The brain-task classification accuracy is 80.47%, which has a 2.67% reduction compared to that without the SAS technique.

CNN-based classification of fNIRS signals in motor imagery BCI system

Objective. Development of a brain-computer interface (BCI) requires classification of brain neural activities to different states. Functional near-infrared spectroscopy (fNIRS) can measure the brain activities and has great potential for BCI. In recent years, a large number of classification algorithms have been proposed, in which deep learning methods, especially convolutional neural network (CNN) methods are successful. fNIRS signal has typical time series properties, we combined fNIRS data and kinds of CNN-based time series classification (TSC) methods to classify BCI task.Approach. In this study, participants were recruited for a left and right hand motor imagery experiment and the cerebral neural activities were recorded by fNIRS equipment (FOIRE-3000). TSC methods are used to distinguish the brain activities when imagining the left or right hand. We have tested the overall person, single person and overall person with single-channel classification results, and these methods achieved excellent classification results. We also compared the CNN-based TSC methods with traditional classification methods such as support vector machine.Main results. Experiments showed that the CNN-based methods have significant advantages in classification accuracy: the CNN-based methods have achieved remarkable results in the classification of left-handed and right-handed imagination tasks, reaching 98.6% accuracy on overall person, 100% accuracy on single person, and in the single-channel classification an accuracy of 80.1% has been achieved with the best-performing channel.Significance. These results suggest that using the CNN-based TSC methods can significantly improve the BCI performance and also lay the foundation for the miniaturization and portability of training rehabilitation equipment.

Fugl-Meyer hand motor imagination recognition for brain–computer interfaces using only fNIRS

As a relatively new physiological signal of brain, functional near-infrared spectroscopy (fNIRS) is being used more and more in brain–computer interface field, especially in the task of motor imagery. However, the classification accuracy based on this signal is relatively low. To improve the accuracy of classification, this paper proposes a new experimental paradigm and only uses fNIRS signals to complete the classification task of six subjects. Notably, the experiment is carried out in a non-laboratory environment, and movements of motion imagination are properly designed. And when the subjects are imagining the motions, they are also subvocalizing the movements to prevent distraction. Therefore, according to the motor area theory of the cerebral cortex, the positions of the fNIRS probes have been slightly adjusted compared with other methods. Next, the signals are classified by nine classification methods, and the different features and classification methods are compared. The results show that under this new experimental paradigm, the classification accuracy of 89.12% and 88.47% can be achieved using the support vector machine method and the random forest method, respectively, which shows that the paradigm is effective. Finally, by selecting five channels with the largest variance after empirical mode decomposition of the original signal, similar classification results can be achieved.

An fNIRS-Based Motor Imagery BCI for ALS: A Subject-Specific Data-Driven Approach

OBJECTIVE

Functional near-infrared spectroscopy (fNIRS) has recently gained momentum in research on motor-imagery (MI)-based brain-computer interfaces (BCIs). However, strikingly, most of the research effort is primarily devoted to enhancing fNIRS-based BCIs for healthy individuals. The ability of patients with amyotrophic lateral sclerosis (ALS), among the main BCI end-users to utilize fNIRS-based hemodynamic responses to efficiently control an MI-based BCI, has not yet been explored. This study aims to quantify subject-specific spatio-temporal characteristics of ALS patients' hemodynamic responses to MI tasks, and to investigate the feasibility of using these responses as a means of communication to control a binary BCI.

METHODS

Hemodynamic responses were recorded using fNIRS from eight patients with ALS while performing MI-Rest tasks. The generalized linear model (GLM) analysis was conducted to statistically estimate and evaluate individualized spatial activation. Selected channel sets were statistically optimized for classification. Subject-specific discriminative features, including a proposed data-driven estimated coefficient obtained from GLM, and optimized classification parameters were identified and used to further evaluate the performance using a linear support vector machine (SVM) classifier.

RESULTS

Inter-subject variations were observed in spatio-temporal characteristics of patients' hemodynamic responses. Using optimized classification parameters and feature sets, all subjects could successfully use their MI hemodynamic responses to control a BCI with an average classification accuracy of 85.4% ± 9.8%.

SIGNIFICANCE

Our results indicate a promising application of fNIRS-based MI hemodynamic responses to control a binary BCI by ALS patients. These findings highlight the importance of subject-specific data-driven approaches for identifying discriminative spatio-temporal characteristics for an optimized BCI performance.