Exploration of User's Mental State Changes during Performing Brain-Computer Interface

Combination of Channel Reordering Strategy and Dual CNN-LSTM for Epileptic Seizure Prediction Using Three iEEG Datasets

OBJECTIVE

Intracranial electroencephalogram (iEEG) signals are generally recorded using multiple channels, and channel selection is therefore a significant means in studying iEEG-based seizure prediction. For n channels, [Formula: see text] channel cases can be generated for selection. However, by this means, an increase in n can cause an exponential increase in computational consumption, which may result in a failure of channel selection when n is too large. Hence, it is necessary to explore reasonable channel selection strategies under the premise of controlling computational consumption and ensuring high classification accuracy. Given this, we propose a novel method of channel reordering strategy combined with dual CNN-LSTM for effectively predicting seizures.

METHOD

First, for each patient with n channels, interictal and preictal iEEG samples from each single channel are input into the CNN-LSTM model for classification. Then, the F1-score of each single channel is calculated, and the channels are reordered in descending order according to the size of F1-scores (channel reordering strategy). Next, iEEG signals with an increasing number of channels are successively fed into the CNN-LSTM model for classification again. Finally, according to the classification results from n channel cases, the channel case with the highest classification rate is selected.

RESULTS

Our method is evaluated on the three iEEG datasets: the Freiburg, the SWEC-ETHZ and the American Epilepsy Society Seizure Prediction Challenge (AES-SPC). At the event-based level, the sensitivities of 100%, 100% and 90.5%, and the false prediction rates (FPRs) of 0.10/h, 0/h and 0.47/h, are achieved for the three datasets, respectively. Moreover, compared to an unspecific random predictor, our method also shows a better performance for all patients and dogs from the three datasets. At the segment-based level, the sensitivities-specificities-accuracies-AUCs of 88.1%-94.0%-93.5%-0.9101, 99.1%-99.7%-99.6%-0.9935, and 69.2%-79.9%-78.2%-0.7373, are attained for the three datasets, respectively.

CONCLUSION

Our method can effectively predict seizures and address the challenge of an excessive number of channels during channel selection.

A new approach for neural decoding by inspiring of hyperdimensional computing for implantable intra-cortical BMIs

In the field of Brain Machine Interface (BMI), the process of translating motor intention into a machine command is denoted as decoding. However, despite recent advancements, decoding remains a formidable challenge within BMI. The utilization of current decoding algorithms in the field of BMI often involves computational complexity and requires the use of computers. This is primarily due to the reliance on mathematical models to address the decoding issue and perform subsequent output calculations. Unfortunately, computers are not feasible for implantable BMI systems due to their size and power consumption. To address this predicament, this study proposes a pioneering approach inspired by hyperdimensional computing. This approach first involves identifying the pattern of each stimulus by considering the normal firing rate distribution of each neuron. Subsequently, the newly observed firing pattern for each input is compared with the patterns detected at each moment for each neuron. The algorithm, which shares similarities with hyperdimensional computing, identifies the most similar pattern as the final output. This approach reduces the dependence on mathematical models. The efficacy of this method is assessed through the utilization of an authentic dataset acquired from the Frontal Eye Field (FEF) of two male rhesus monkeys. The output space encompasses eight possible angles. The results demonstrate an accuracy rate of 51.5% while exhibiting significantly low computational complexity, involving a mere 2050 adder operators. Furthermore, the proposed algorithm is implemented on a field-programmable gate array (FPGA) and as an ASIC designe in a standard CMOS 180 nm technology, underscoring its suitability for real-time implantable BMI applications. The implementation required only 2.3 Kbytes of RAM, occupied an area of 2.2 mm2, and consumed 9.32 µW at a 1.8 V power supply. Consequently, the proposed solution represents an accurate, low computational complexity, hardware-friendly, and real-time approach.

Considerate motion imagination classification method using deep learning

In order to improve the classification accuracy of motion imagination, a considerate motion imagination classification method using deep learning is proposed. Specifically, based on a graph structure suitable for electroencephalography as input, the proposed model can accurately represent the distribution of electroencephalography electrodes in non-Euclidean space and fully consider the spatial correlation between electrodes. In addition, the spatial-spectral-temporal multi-dimensional feature information was extracted from the spatial-temporal graph representation and spatial-spectral graph representation transformed from the original electroencephalography signal using the dual branch architecture. Finally, the attention mechanism and global feature aggregation module were designed and combined with graph convolution to adaptively capture the dynamic correlation intensity and effective feature of electroencephalography signals in various dimensions. A series of contrast experiments and ablation experiments on several different public brain-computer interface datasets demonstrated that the excellence of proposed method. It is worth mentioning that, the proposed model is a general framework for the classification of electroencephalography signals, which is suitable for emotion recognition, sleep staging and other fields based on electroencephalography research. Moreover, the model has the potential to be applied in the medical field of motion imagination rehabilitation in real life.

Cross-Platform Implementation of an SSVEP-Based BCI for the Control of a 6-DOF Robotic Arm

Robotics has been successfully applied in the design of collaborative robots for assistance to people with motor disabilities. However, man-machine interaction is difficult for those who suffer severe motor disabilities. The aim of this study was to test the feasibility of a low-cost robotic arm control system with an EEG-based brain-computer interface (BCI). The BCI system relays on the Steady State Visually Evoked Potentials (SSVEP) paradigm. A cross-platform application was obtained in C++. This C++ platform, together with the open-source software Openvibe was used to control a Stäubli robot arm model TX60. Communication between Openvibe and the robot was carried out through the Virtual Reality Peripheral Network (VRPN) protocol. EEG signals were acquired with the 8-channel Enobio amplifier from Neuroelectrics. For the processing of the EEG signals, Common Spatial Pattern (CSP) filters and a Linear Discriminant Analysis classifier (LDA) were used. Five healthy subjects tried the BCI. This work allowed the communication and integration of a well-known BCI development platform such as Openvibe with the specific control software of a robot arm such as Stäubli TX60 using the VRPN protocol. It can be concluded from this study that it is possible to control the robotic arm with an SSVEP-based BCI with a reduced number of dry electrodes to facilitate the use of the system.

Associations between Heart Rate Variability and Brain Activity during a Working Memory Task: A Preliminary Electroencephalogram Study on Depression and Anxiety Disorder

Heart rate variability (HRV) has been suggested to reflect executive function and related neural activity. Executive dysfunction has been suggested to play an important role in the pathophysiology of emotional disorders. The purpose of this study was to investigate whether HRV showed a significant correlation with electroencephalogram (EEG) during a working memory performance in patients with depressive or anxiety disorder. A retrospective analysis was conducted with data from 61 patients with depressive disorder (43 women and 18 men) and 59 patients with anxiety disorder (35 women and 24 men). HRV was measured in the resting state, and EEG was recorded in the resting state and during the execution of a working memory task. It was performed in patients with depressive and anxiety disorder, and the paired sample t-test between resting state and task performance, as well as the partial correlation analysis between HRV and EEG, was conducted. Both depressed and anxious patients showed weaker beta relative power during the working memory task compared to the rest period. The resting-state EEG did not correlate with HRV parameters in both groups. In depressed patients, HRV showed a positive correlation with delta power during the task and a negative correlation with beta relative power during the task. In patients with anxiety disorder, HRV showed a significant positive correlation with theta power of the right frontal region during the task. Our results suggest that HRV would be related to executive-function-related neural activity in patients with depressive or anxiety disorder. Future studies with more subjects, including healthy controls, are needed to verify the correlation between HRV and EEG and to come up with a more comprehensive picture of neurobiological changes in emotional disorders.

Neurophysiological Verbal Working Memory Patterns in Children: Searching for a Benchmark of Modality Differences in Audio/Video Stimuli Processing

Exploration of specific brain areas involved in verbal working memory (VWM) is a powerful but not widely used tool for the study of different sensory modalities, especially in children. In this study, for the first time, we used electroencephalography (EEG) to investigate neurophysiological similarities and differences in response to the same verbal stimuli, expressed in the auditory and visual modality during the n-back task with varying memory load in children. Since VWM plays an important role in learning ability, we wanted to investigate whether children elaborated the verbal input from auditory and visual stimuli through the same neural patterns and if performance varies depending on the sensory modality. Performance in terms of reaction times was better in visual than auditory modality (p = 0.008) and worse as memory load increased regardless of the modality (p < 0.001). EEG activation was proportionally influenced by task level and was evidenced in theta band over the prefrontal cortex (p = 0.021), along the midline (p = 0.003), and on the left hemisphere (p = 0.003). Differences in the effects of the two modalities were seen only in gamma band in the parietal cortices (p = 0.009). The values of a brainwave-based engagement index, innovatively used here to test children in a dual-modality VWM paradigm, varied depending on n-back task level (p = 0.001) and negatively correlated (p = 0.002) with performance, suggesting its computational effectiveness in detecting changes in mental state during memory tasks involving children. Overall, our findings suggest that auditory and visual VWM involved the same brain cortical areas (frontal, parietal, occipital, and midline) and that the significant differences in cortical activation in theta band were more related to memory load than sensory modality, suggesting that VWM function in the child's brain involves a cross-modal processing pattern.

Anti-Heartbeat-Evoked Potentials Performance in Event-Related Potentials-Based Mental Workload Assessment

The aim of this study was to determine the effect of heartbeat-evoked potentials (HEPs) on the performance of an event-related potential (ERP)-based classification of mental workload (MWL). We produced low- and high-MWLs using a mental arithmetic task and measured the ERP response of 14 participants. ERP trials were divided into three conditions based on the effect of HEPs on ERPs: ERP_HEP, containing the heartbeat in a period of 280–700ms in ERP epochs after the target; ERP_A-HEP, not including the heartbeat within the same period; and ERP_T, all trials including ERP_A-HEP and ERP_HEP. We then compared MWL classification performance using the amplitude and latency of the P600 ERP among the three conditions. The ERP_A-HEP condition achieved an accuracy of 100% using a radial basis function-support vector machine (with 10-fold cross-validation), showing an increase of 14.3 and 28.6% in accuracy compared to ERP_T (85.7%) and ERP_HEP (71.4%), respectively. The results suggest that evoked potentials caused by heartbeat overlapped or interfered with the ERPs and weakened the ERP response to stimuli. This study reveals the effect of the evoked potentials induced by heartbeats on the performance of the MWL classification based on ERPs.

Cross-Subject Zero Calibration Driver's Drowsiness Detection: Exploring Spatiotemporal Image Encoding of EEG Signals for Convolutional Neural Network Classification

This paper explores two methodologies for drowsiness detection using EEG signals in a sustained-attention driving task considering pre-event time windows, and focusing on cross-subject zero calibration. Driving accidents are a major cause of injuries and deaths on the road. A considerable portion of those are due to fatigue and drowsiness. Advanced driver assistance systems that could detect mental states which are associated with hazardous situations, such as drowsiness, are of critical importance. EEG signals are used widely for brain-computer interfaces, as well as mental state recognition. However, these systems are still difficult to design due to very low signal-to-noise ratios and cross-subject disparities, requiring individual calibration cycles. To tackle this research domain, here, we explore drowsiness detection based on EEG signals' spatiotemporal image encoding representations in the form of either recurrence plots or gramian angular fields for deep convolutional neural network (CNN) classification. Results comparing both techniques using a public dataset of 27 subjects show a superior balanced accuracy of up to 75.87% for leave-one-out cross-validation, using both techniques, against works in the literature, demonstrating the possibility to pursue cross-subject zero calibration design.

Vector Phase Analysis Approach for Sleep Stage Classification: A Functional Near-Infrared Spectroscopy-Based Passive Brain-Computer Interface

A passive brain-computer interface (BCI) based upon functional near-infrared spectroscopy (fNIRS) brain signals is used for earlier detection of human drowsiness during driving tasks. This BCI modality acquired hemodynamic signals of 13 healthy subjects from the right dorsolateral prefrontal cortex (DPFC) of the brain. Drowsiness activity is recorded using a continuous-wave fNIRS system and eight channels over the right DPFC. During the experiment, sleep-deprived subjects drove a vehicle in a driving simulator while their cerebral oxygen regulation (CORE) state was continuously measured. Vector phase analysis (VPA) was used as a classifier to detect drowsiness state along with sleep stage-based threshold criteria. Extensive training and testing with various feature sets and classifiers are done to justify the adaptation of threshold criteria for any subject without requiring recalibration. Three statistical features (mean oxyhemoglobin, signal peak, and the sum of peaks) along with six VPA features (trajectory slopes of VPA indices) were used. The average accuracies for the five classifiers are 90.9% for discriminant analysis, 92.5% for support vector machines, 92.3% for nearest neighbors, 92.4% for both decision trees, and ensembles over all subjects' data. Trajectory slopes of CORE vector magnitude and angle: m(|R|) and m(∠R) are the best-performing features, along with ensemble classifier with the highest accuracy of 95.3% and minimum computation time of 40 ms. The statistical significance of the results is validated with a p-value of less than 0.05. The proposed passive BCI scheme demonstrates a promising technique for online drowsiness detection using VPA along with sleep stage classification.

Data Augmentation: Using Channel-Level Recombination to Improve Classification Performance for Motor Imagery EEG

The quality and quantity of training data are crucial to the performance of a deep-learning-based brain-computer interface (BCI) system. However, it is not practical to record EEG data over several long calibration sessions. A promising time- and cost-efficient solution is artificial data generation or data augmentation (DA). Here, we proposed a DA method for the motor imagery (MI) EEG signal called brain-area-recombination (BAR). For the BAR, each sample was first separated into two ones (named half-sample) by left/right brain channels, and the artificial samples were generated by recombining the half-samples. We then designed two schemas (intra- and adaptive-subject schema) corresponding to the single- and multi-subject scenarios. Extensive experiments using the classifier of EEGnet were conducted on two public datasets under various training set sizes. In both schemas, the BAR method can make the EEGnet have a better performance of classification (p < 0.01). To make a comparative investigation, we selected two common DA methods (noise-added and flipping), and the BAR method beat them (p < 0.05). Further, using the proposed BAR for augmentation, EEGnet achieved up to 8.3% improvement than a typical decoding algorithm CSP-SVM (p < 0.01), note that both the models were trained on the augmented dataset. This study shows that BAR usage can significantly improve the classification ability of deep learning to MI-EEG signals. To a certain extent, it may promote the development of deep learning technology in the field of BCI.

Evaluating the Effect of Stimuli Color and Frequency on SSVEP

Brain-computer interfaces (BCI) can extract information about the subject's intentions by registering and processing electroencephalographic (EEG) signals to generate actions on physical systems. Steady-state visual-evoked potentials (SSVEP) are produced when the subject stares at flashing visual stimuli. By means of spectral analysis and by measuring the signal-to-noise ratio (SNR) of its harmonic contents, the observed stimulus can be identified. Stimulus color matters, and some authors have proposed red because of its ability to capture attention, while others refuse it because it might induce epileptic seizures. Green has also been proposed and it is claimed that white may generate the best signals. Regarding frequency, middle frequencies are claimed to produce the best SNR, although high frequencies have not been thoroughly studied, and might be advantageous due to the lower spontaneous cerebral activity in this frequency band. Here, we show white, red, and green stimuli, at three frequencies: 5 (low), 12 (middle), and 30 (high) Hz to 42 subjects, and compare them in order to find which one can produce the best SNR. We aim to know if the response to white is as strong as the one to red, and also if the response to high frequency is as strong as the one triggered by lower frequencies. Attention has been measured with the Conner's Continuous Performance Task version 2 (CPT-II) task, in order to search for a potential relationship between attentional capacity and the SNR previously obtained. An analysis of variance (ANOVA) shows the best SNR with the middle frequency, followed by the low, and finally the high one. White gives as good an SNR as red at 12 Hz and so does green at 5 Hz, with no differences at 30 Hz. These results suggest that middle frequencies are preferable and that using the red color can be avoided. Correlation analysis also show a correlation between attention and the SNR at low frequency, so suggesting that for the low frequencies, more attentional capacity leads to better results.

Hybrid Method of Automated EEG Signals' Selection Using Reversed Correlation Algorithm for Improved Classification of Emotions

Based on the growing interest in encephalography to enhance human-computer interaction (HCI) and develop brain-computer interfaces (BCIs) for control and monitoring applications, efficient information retrieval from EEG sensors is of great importance. It is difficult due to noise from the internal and external artifacts and physiological interferences. The enhancement of the EEG-based emotion recognition processes can be achieved by selecting features that should be taken into account in further analysis. Therefore, the automatic feature selection of EEG signals is an important research area. We propose a multistep hybrid approach incorporating the Reversed Correlation Algorithm for automated frequency band-electrode combinations selection. Our method is simple to use and significantly reduces the number of sensors to only three channels. The proposed method has been verified by experiments performed on the DEAP dataset. The obtained effects have been evaluated regarding the accuracy of two emotions-valence and arousal. In comparison to other research studies, our method achieved classification results that were 4.20-8.44% greater. Moreover, it can be perceived as a universal EEG signal classification technique, as it belongs to unsupervised methods.

SSVEP-assisted RSVP Brain-Computer Interface paradigm for multi-target classification

Brain-Computer Interface (BCI) is actively involved in optimizing the communication medium between the human brain and external devices.

OBJECTIVE

Rapid serial visual presentation (RSVP) is a robust and highly efficient BCI technique in recognizing target objects but suffers from limited target selections. The BCI systems that combine steady-state visual evoked potential (SSVEP) and RSVP can mitigate this limitation and allow users to operate on multiple targets.

APPROACH

This study proposes a novel SSVEP-assisted RSVP BCI model to improve the performance of classifying the target/non-target objects in a multi-target scenario. In this paradigm, SSVEP stimuli helps in identifying the user's focus location and RSVP stimuli that elicits event-related potentials (ERPs) differentiate target and non-target objects.

MAIN RESULTS

The proposed model achieved an offline accuracy of 81.59% by using 12 electroencephalogram (EEG) channels and an online (real-time) accuracy of 78.10% when only 4 EEG channels are considered. Further, the biomarkers of physiological states are analyzed to assess the cognitive states (mental fatigue and user attention) of the participants based on resting theta and alpha band powers. The results indicate an inverse relationship between the BCI performance and the resting EEG power, validating that the subjects' performance is affected by physiological states for prolonged BCI tasks.

SIGNIFICANCE

Our findings demonstrate that the combination of SSVEP and RSVP stimuli improves the BCI performance and further enhances the possibility of performing multiple user command tasks, which are inevitable in real-world applications. Additionally, the cognitive state biomarkers discussed imply the need for an efficient and attractive experimental paradigm that reduces the physiological state disparities and provide enhanced BCI performance.

Global Neural Activities Changes under Human Inhibitory Control Using Translational Scenario

This study presents a new approach to exploring human inhibition in a realistic scenario. In previous inhibition studies, the stimulus design of go/no-go task generally used a simple symbol for the go and stop signals. We can understand the neural activity of inhibition through simple symbol scenario. In the real world, situations of human inhibition are more complex than performing an experiment in the laboratory scale. How to explore the neural activities of inhibition in a realistic environment is more complex. Consequently, we designed a battlefield scenario to investigate the neural activities of inhibition in a more realistic environmental setting. The battlefield scenario provides stronger emotion, motivation and real-world experiences for participants during inhibition. In the battlefield scenario, the signs of fixation, go and stop were replaced by images of a sniper scope, a target and a non-target. The battlefield scenario is a shooting game between the enemy and the soldiers. In battlefield scenario participants played the role of the soldiers for shooting target and to stop shooting when a non-target appeared. Electroencephalography (EEG) signals from twenty participants were acquired and analyzed using independent component analysis (ICA) and dipole source localization method. The results of event-related potential (ERP) showed a significant modulation of the peaks N1, N2 and P3 in the frontal and cingulate cortices under inhibitory control. The partially overlapping ERP N2 and P3 waves were associated with inhibition in the frontal cortex. The ERP N2, N1 and P3 waves in the cingulate cortex are related to sustained attention, motivation, emotion and inhibitory control. In addition, the event-related spectral perturbation (ERSP) results shows that the powers of the delta and theta bands increased significantly in the frontal and cingulate cortices under human inhibitory control. The EEG-ERP waves and power spectra in the frontal and cingulate cortices were found more increased than in the parietal, occipital, left and right motor cortices after successful stop. These findings provide new insights to understand the global neural activities changes during human inhibitory control with realistic environmental scenario.