Intersession Instability in fNIRS-Based Emotion Recognition

Research on Emotion Recognition Method of Cerebral Blood Oxygen Signal Based on CNN-Transformer Network

In recent years, research on emotion recognition has become more and more popular, but there are few studies on emotion recognition based on cerebral blood oxygen signals. Since the electroencephalogram (EEG) is easily disturbed by eye movement and the portability is not high, this study uses a more comfortable and convenient functional near-infrared spectroscopy (fNIRS) system to record brain signals from participants while watching three different types of video clips. During the experiment, the changes in cerebral blood oxygen concentration in the 8 channels of the prefrontal cortex of the brain were collected and analyzed. We processed and divided the collected cerebral blood oxygen data, and used multiple classifiers to realize the identification of the three emotional states of joy, neutrality, and sadness. Since the classification accuracy of the convolutional neural network (CNN) in this research is not significantly superior to that of the XGBoost algorithm, this paper proposes a CNN-Transformer network based on the characteristics of time series data to improve the classification accuracy of ternary emotions. The network first uses convolution operations to extract channel features from multi-channel time series, then the features and the output information of the fully connected layer are input to the Transformer netork structure, and its multi-head attention mechanism is used to focus on different channel domain information, which has better spatiality. The experimental results show that the CNN-Transformer network can achieve 86.7% classification accuracy for ternary emotions, which is about 5% higher than the accuracy of CNN, and this provides some help for other research in the field of emotion recognition based on time series data such as fNIRS.

Emotion recognition in EEG signals using deep learning methods: A review

Emotions are a critical aspect of daily life and serve a crucial role in human decision-making, planning, reasoning, and other mental states. As a result, they are considered a significant factor in human interactions. Human emotions can be identified through various sources, such as facial expressions, speech, behavior (gesture/position), or physiological signals. The use of physiological signals can enhance the objectivity and reliability of emotion detection. Compared with peripheral physiological signals, electroencephalogram (EEG) recordings are directly generated by the central nervous system and are closely related to human emotions. EEG signals have the great spatial resolution that facilitates the evaluation of brain functions, making them a popular modality in emotion recognition studies. Emotion recognition using EEG signals presents several challenges, including signal variability due to electrode positioning, individual differences in signal morphology, and lack of a universal standard for EEG signal processing. Moreover, identifying the appropriate features for emotion recognition from EEG data requires further research. Finally, there is a need to develop more robust artificial intelligence (AI) including conventional machine learning (ML) and deep learning (DL) methods to handle the complex and diverse EEG signals associated with emotional states. This paper examines the application of DL techniques in emotion recognition from EEG signals and provides a detailed discussion of relevant articles. The paper explores the significant challenges in emotion recognition using EEG signals, highlights the potential of DL techniques in addressing these challenges, and suggests the scope for future research in emotion recognition using DL techniques. The paper concludes with a summary of its findings.

Cognitive and motor cortex activation during robot-assisted multi-sensory interactive motor rehabilitation training: An fNIRS based pilot study

Objective

This study aimed to evaluate the effects of multiple virtual reality (VR) interaction modalities based on force-haptic feedback combined with visual or auditory feedback in different ways on cerebral cortical activation by functional near-infrared spectroscopy (fNIRS). Methods: A modular multi-sensory VR interaction system based on a planar upper-limb rehabilitation robot was developed. Twenty healthy participants completed active elbow flexion and extension training in four VR interaction patterns, including haptic (H), haptic + auditory (HA), haptic + visual (HV), and haptic + visual + auditory (HVA). Cortical activation changes in the sensorimotor cortex (SMC), premotor cortex (PMC), and prefrontal cortex (PFC) were measured.

Results

Four interaction patterns all had significant activation effects on the motor and cognitive regions of the cerebral cortex (p < 0.05). Among them, in the HVA interaction mode, the cortical activation of each ROI was the strongest, followed by HV, HA, and H. The connectivity between channels of SMC and bilateral PFC, as well as the connectivity between channels in PMC, was the strongest under HVA and HV conditions. Besides, the two-way ANOVA of visual and auditory feedback showed that it was difficult for auditory feedback to have a strong impact on activation without visual feedback. In addition, under the condition of visual feedback, the effect of fusion auditory feedback on the activation degree was significantly higher than that of no auditory feedback.

Conclusions

The interaction mode of visual, auditory, and haptic multi-sensory integration is conducive to stronger cortical activation and cognitive control. Besides, there is an interaction effect between visual and auditory feedback, thus improving the cortical activation level. This research enriches the research on activation and connectivity of cognitive and motor cortex in the process of modular multi-sensory interaction training of rehabilitation robots. These conclusions provide a theoretical basis for the optimal design of the interaction mode of the rehabilitation robot and the possible scheme of clinical VR rehabilitation.

Impacts of simplifying articulation movements imagery to speech imagery BCI performance

Objective.Speech imagery (SI) can be used as a reliable, natural, and user-friendly activation task for the development of brain-computer interface (BCI), which empowers individuals with severe disabilities to interact with their environment. The functional near-infrared spectroscopy (fNIRS) is advanced as one of the most suitable brain imaging methods for developing BCI systems owing to its advantages of being non-invasive, portable, insensitive to motion artifacts, and having relatively high spatial resolution.Approach.To improve the classification performance of SI BCI based on fNIRS, a novel paradigm was developed in this work by simplifying the articulation movements in SI to make the articulation movement differences clearer between different words imagery tasks. A SI BCI was proposed to directly answer questions by covertly rehearsing the word '' or '' ('yes' or 'no' in English), and an unconstrained rest task also was contained in this BCI. The articulation movements of SI were simplified by retaining only the movements of the jaw and lips of vowels in Chinese Pinyin for words '' and ''.Main results.Compared with conventional speech imagery, simplifying the articulation movements in SI could generate more different brain activities among different tasks, which led to more differentiable temporal features and significantly higher classification performance. The average 3-class classification accuracies of the proposed paradigm across all 20 participants reached 69.6% and 60.2% which were about 10.8% and 5.6% significantly higher than those of the conventional SI paradigm operated in the 0-10 s and 0-2.5 s time windows, respectively.Significance.These results suggested that simplifying the articulation movements in SI is promising for improving the classification performance of intuitive BCIs based on speech imagery.

fNIRS Evidence for Distinguishing Patients With Major Depression and Healthy Controls

In recent years, major depressive disorder (MDD) has been shown to negatively impact physical recovery in a variety of patients. Functional near-infrared spectroscopy (fNIRS) is a tool that can potentially supplement clinical interviews and mental state examinations to establish a psychiatric diagnosis and monitor treatment progress. Thirty-two subjects, including 16 patients clinically diagnosed with MDD and 16 healthy controls (HCs), participated in the study. Brain oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) responses were recorded using a 22-channel continuous-wave fNIRS device while the subjects performed the emotional sound test. This study evaluated the difference between MDD patients and HCs using a variety of methods. In a comparison of the Pearson correlation coefficients between the HbO/HbR responses of each fNIRS channel and four scores, MDD patients and HCs had significantly different Athens Insomnia Scale (AIS) scores. By quantitative evaluation of the functional association, we found that MDD patients had aberrant functional connectivity compared with HCs. Furthermore, we concluded that compared with HCs, there were marked abnormalities in blood oxygen in the bilateral ventrolateral prefrontal cortex (VLPFC) and bilateral dorsolateral prefrontal cortex (DLPFC). Four statistical-based features extracted from HbO signals and four vector-based features from both HbO and HbR served as inputs to four simple neural networks (multilayer neural network (MNN), feedforward neural network (FNN), cascade forward neural network (CFNN) and recurrent neural network (RNN)). Through an analysis of combinations of different features, the combination of 4 common features (mean, STD, area under the receiver operating characteristic curve (AUC) and slope) yielded the highest classification accuracy of 89.74% for fear emotion. The combination of four novel feature (CBV, COE, |L | and K) resulted in a classification accuracy of 99.94% for fear emotion. The top 10 common and novel features were selected by the ReliefF feature selection algorithm, resulting in classification accuracies of 83.52% and 91.99%, respectively. This study identified the AUC and angle K as specific neuromarkers for predicting MDD across specific depression-related regions of the prefrontal cortex (PFC). These findings suggest that the fNIRS measurement of the PFC may serve as a supplementary test in routine clinical practice to further support a diagnosis of MDD.

A systematic review of studies that used NIRS to measure neural activation during emotion processing in healthy individuals

Functional neuroimaging provides an avenue for earlier diagnosis and tailored treatment of psychological disorders characterised by emotional impairment. Near-infrared spectroscopy (NIRS) offers ecological advantages compared to other neuroimaging techniques and suitability of measuring regions involved in emotion functions. A systematic review was conducted to evaluate the capacity of NIRS to detect activation during emotion processing and to provide recommendations for future research. Following a comprehensive literature search, we reviewed 85 journal articles, which compared activation during emotional experience, regulation or perception with either a neutral condition or baseline period among healthy participants. The quantitative synthesis of outcomes was limited to thematical analysis, owing to the lack of standardisation between studies. Although most studies found increased prefrontal activity during emotional experience and regulation, the findings were more inconsistent for emotion perception. Some researchers reported increased activity during the task, some reported decreases, some no significant changes, and some reported mixed findings depending on the valence and region. We propose that variations in the cognitive task and stimuli, recruited sample, and measurement and analysis of data are the primary causes of inconsistency. Recommendations to improve consistency in future research by carefully considering the choice of population, cognitive task and analysis approach are provided.

Sensors and Systems for Physical Rehabilitation and Health Monitoring-A Review

The use of wearable equipment and sensing devices to monitor physical activities, whether for well-being, sports monitoring, or medical rehabilitation, has expanded rapidly due to the evolution of sensing techniques, cheaper integrated circuits, and the development of connectivity technologies. In this scenario, this paper presents a state-of-the-art review of sensors and systems for rehabilitation and health monitoring. Although we know the increasing importance of data processing techniques, our focus was on analyzing the implementation of sensors and biomedical applications. Although many themes overlap, we organized this review based on three groups: Sensors in Healthcare, Home Medical Assistance, and Continuous Health Monitoring; Systems and Sensors in Physical Rehabilitation; and Assistive Systems.

Enhanced Accuracy for Multiclass Mental Workload Detection Using Long Short-Term Memory for Brain-Computer Interface

Cognitive workload is one of the widely invoked human factors in the areas of human-machine interaction (HMI) and neuroergonomics. The precise assessment of cognitive and mental workload (MWL) is vital and requires accurate neuroimaging to monitor and evaluate the cognitive states of the brain. In this study, we have decoded four classes of MWL using long short-term memory (LSTM) with 89.31% average accuracy for brain-computer interface (BCI). The brain activity signals are acquired using functional near-infrared spectroscopy (fNIRS) from the prefrontal cortex (PFC) region of the brain. We performed a supervised MWL experimentation with four varying MWL levels on 15 participants (both male and female) and 10 trials of each MWL per participant. Real-time four-level MWL states are assessed using fNIRS system, and initial classification is performed using three strong machine learning (ML) techniques, support vector machine (SVM), k-nearest neighbor (k-NN), and artificial neural network (ANN) with obtained average accuracies of 54.33, 54.31, and 69.36%, respectively. In this study, novel deep learning (DL) frameworks are proposed, which utilizes convolutional neural network (CNN) and LSTM with 87.45 and 89.31% average accuracies, respectively, to solve high-dimensional four-level cognitive states classification problem. Statistical analysis, t-test, and one-way F-test (ANOVA) are also performed on accuracies obtained through ML and DL algorithms. Results show that the proposed DL (LSTM and CNN) algorithms significantly improve classification performance as compared with ML (SVM, ANN, and k-NN) algorithms.

Functional near-infrared spectroscopy for intraoperative brain mapping

Functional near-infrared spectroscopy (fNIRS) is a relatively new seizure-free technique and its value for intraoperative brain mapping is unknown. We examine the feasibility of fNIRS for intraoperative functional brain mapping. A 1 × 1    cm 2 density fNIRS probe specially designed for intraoperative use was used to map brain function in adult patients undergoing awake brain surgery and performing motor and/or language tasks. The ability of fNIRS for functional mapping was compared with direct cortical stimulation (DCS) and regression was used to determine if mean blood pressure (MBP) and blood hemoglobin influenced fNIRS measurements. Eighteen patients underwent awake craniotomy and performed 19 language- and 17 motor-related tasks. fNIRS mapping was highly correlated with DCS for 10 language- and 7 motor-related tasks. fNIRS was able to detect functional language ( p < 0.001 ) and motor areas ( p = 0.002 ). Compared to DCS, fNIRS was less accurate in determining both functional language (at least 22.64%, p < 0.001 ) and motor areas (at least 32.74%, p < 0.001 ). Higher MBP and blood hemoglobin were associated with better fNIRS results ( p = 0.045 and 0.007, respectively). No seizures or other complications occurred during fNIRS measurement. fNIRS is a promising seizure-free technique for intraoperative brain mapping. The accuracy of current technology needs further development for clinical use.