Mental fatigue level detection based on event related and visual evoked potentials features fusion in virtual indoor environment

SSVEP unsupervised adaptive feature recognition method based on self-similarity of same-frequency signals

Introduction

As an important human-computer interaction technology, steady-state visual evoked potential (SSVEP) plays a key role in the application of brain computer interface (BCI) systems by accurately decoding SSVEP signals. Currently, the majority SSVEP feature recognition methods use a static classifier. However, electroencephalogram (EEG) signals are non-stationary and time-varying. Hence, an adaptive classification method would be an alternative option to a static classifier for tracking the changes in EEG feature distribution, as its parameters can be re-estimated and updated with the input of new EEG data.

Methods

In this study, an unsupervised adaptive classification algorithm is designed based on the self-similarity of same-frequency signals. The proposed classification algorithm saves the EEG data that has undergone feature recognition as a template signal in accordance with its estimated label, and the new testing signal is superimposed with the template signals at each stimulus frequency as the new test signals to be analyzed. With the continuous input of EEG data, the template signals are continuously updated.

Results

By comparing the classification accuracy of the original testing signal and the testing signal superimposed with the template signals, this study demonstrates the effectiveness of using the self-similarity of same-frequency signals in the adaptive classification algorithm. The experimental results also show that the longer the SSVEP-BCI system is used, the better the responses of users on SSVEP are, and the more significantly the adaptive classification algorithm performs in terms of feature recognition. The testing results of two public datasets show that the adaptive classification algorithm outperforms the static classification method in terms of feature recognition.

Discussion

The proposed adaptive classification algorithm can update the parameters with the input of new EEG data, which is of favorable impact for the accurate analysis of EEG data with time-varying characteristics.

Evaluation of Diurnal Changes of Mental Fatigue Using a New Portable Device for Visual Cognitive Evoked Potentials

In the age homogenous group of 13 healthy volunteers, we examined visual evoked potentials (VEP) visually evoked cognitive potentials (event-related potentials - ERP) and choice reaction time (CRT) five times during the day (from 10.00 a.m. up to midnight) to verify whether there are significant changes of the measured parameters of the cortical evoked potentials and CRT which might reflect the level of the mental fatigue. The electrophysiological testing was done with the use of a new portable VEP device named "VEPpeak" enabling to perform the examination outside standard labs in almost any conditions. It was found that the latency of ERP (P300 peak time) and CRT displayed significant prolongation toward midnight while VEP latency and all amplitudes did not change significantly. This pilot study supports our idea that the portable VEP device possibly might be used for the objective examination of mental fatigue that is needed in many situations. This should be confirmed in a larger study also including a comparison with non-electrophysiological fatigue testing.

EEG-based mental fatigue detection using linear prediction cepstral coefficients and Riemann spatial covariance matrix

Objective. Establishing a mental fatigue monitoring system is of great importance as for severe fatigue may cause unimaginable consequences. Electroencephalogram (EEG) is often utilized for mental fatigue detection because of its high temporal resolution and ease of use. However, many EEG-based approaches for detecting mental fatigue only take into account the feature extraction of a single domain and do not fully exploit the information that EEG may offer.Approach. In our work, we propose a new algorithm for mental fatigue detection based on multi-domain feature extraction and fusion. EEG components representing fatigue are closely related in the past and present because fatigue is a dynamic and gradual process. Accordingly, the idea of linear prediction is used to fit the current value with a set of sample values in the past to calculate the linear prediction cepstral coefficients (LPCCs) as the time domain feature. Moreover, in order to better capture fatigue-related spatial domain information, the spatial covariance matrix of the original EEG signal is projected into the Riemannian tangent space using the Riemannian geometric method. Then multi-domain features are fused to obtain comprehensive spatio-temporal information.Main results. Experimental results prove the suggested algorithm outperforms existing state-of-the-art methods, achieving an average accuracy of 87.10% classification on the public dataset SEED-VIG (three categories) and 97.40% classification accuracy (two categories) on the dataset made by self-designed experiments.Significance. These findings show that our proposed strategy perform more effectively for mental fatigue detection based on EEG.

A multi-modal brain-computer interface based on threshold discrimination and its application in wheelchair control

In this study, we propose a novel multi-modal brain-computer interface (BCI) system based on the threshold discrimination, which is proposed for the first time to distinguish between SSVEP and MI potentials. The system combines these two heterogeneous signals to increase the number of control commands and improve the performance of asynchronous control of external devices. In this research, an electric wheelchair is controlled as an example. The user can continuously control the wheelchair to turn left/right through motion imagination (MI) by imagining left/right-hand movement and generate another 6 commands for the wheelchair control by focusing on the SSVEP stimulation panel. Ten subjects participated in a MI training session and eight of them completed a mobile obstacle-avoidance experiment in a complex environment requesting high control accuracy for successful manipulation. Comparing with the single-modal BCI-controlled wheelchair system, the results demonstrate that the proposed multi-modal method is effective by providing more satisfactory control accuracy, and show the potential of BCI-controlled systems to be applied in complex daily tasks.

Wearable Flexible Electronics Based Cardiac Electrode for Researcher Mental Stress Detection System Using Machine Learning Models on Single Lead Electrocardiogram Signal

In the modern world, wearable smart devices are continuously used to monitor people's health. This study aims to develop an automatic mental stress detection system for researchers based on Electrocardiogram (ECG) signals from smart T-shirts using machine learning classifiers. We used 20 subjects, including 10 from mental stress (after twelve hours of continuous work in the laboratory) and 10 from normal (after completing the sleep or without any work). We also applied three scoring techniques: Chalder Fatigue Scale (CFS), Specific Fatigue Scale (SFS), Depression, Anxiety, and Stress Scale (DASS), to confirm the mental stress. The total duration of ECG recording was 1800 min, including 1200 min during mental stress and 600 min during normal. We calculated two types of features, such as demographic and extracted by ECG signal. In addition, we used Decision Tree (DT), Naive Bayes (NB), Random Forest (RF), and Logistic Regression (LR) to classify the intra-subject (mental stress and normal) and inter-subject classification. The DT leave-one-out model has better performance in terms of recall (93.30%), specificity (96.70%), precision (94.40%), accuracy (93.30%), and F1 (93.50%) in the intra-subject classification. Additionally, The classification accuracy of the system in classifying inter-subjects is 94.10% when using a DT classifier. However, our findings suggest that the wearable smart T-shirt based on the DT classifier may be used in big data applications and health monitoring. Mental stress can lead to mitochondrial dysfunction, oxidative stress, blood pressure, cardiovascular disease, and various health problems. Therefore, real-time ECG signals help assess cardiovascular and related risk factors in the initial stage based on machine learning techniques.

Cross-subject spatial filter transfer method for SSVEP-EEG feature recognition

Objective.Steady-state visual evoked potential (SSVEP) is an important control method of the brain-computer interface (BCI) system. The development of an efficient SSVEP feature decoding algorithm is the core issue in SSVEP-BCI. It has been proposed to use user training data to reduce the spontaneous electroencephalogram activity interference on SSVEP response, thereby improving the feature recognition accuracy of the SSVEP signal. Nevertheless, the tedious data collection process increases the mental fatigue of the user and severely affects the applicability of the BCI system.Approach.A cross-subject spatial filter transfer (CSSFT) method that transfer the existing user model with good SSVEP response to the new user test data without collecting any training data from the new user is proposed.Main results.Experimental results demonstrate that the transfer model increases the distinction of the feature discriminant coefficient between the gaze following target and the non-gaze following target and accurately identifies the wrong target in the fundamental algorithm model. The public datasets show that the CSSFT method significantly increases the recognition performance of canonical correlation analysis (CCA) and filter bank CCA. Additionally, when the data used to calculate the transfer model contains one data block only, the CSSFT method retains its effective feature recognition capabilities.Significance.The proposed method requires no tedious data calibration process for new users, provides an effective technical solution for the transfer of the cross-subject model, and has potential application value for promoting the application of the BCI system.

Fatigue Monitoring Through Wearables: A State-of-the-Art Review

The objective measurement of fatigue is of critical relevance in areas such as occupational health and safety as fatigue impairs cognitive and motor performance, thus reducing productivity and increasing the risk of injury. Wearable systems represent highly promising solutions for fatigue monitoring as they enable continuous, long-term monitoring of biomedical signals in unattended settings, with the required comfort and non-intrusiveness. This is a p rerequisite for the development of accurate models for fatigue monitoring in real-time. However, monitoring fatigue through wearable devices imposes unique challenges. To provide an overview of the current state-of-the-art in monitoring variables associated with fatigue via wearables and to detect potential gaps and pitfalls in current knowledge, a systematic review was performed. The Scopus and PubMed databases were searched for articles published in English since 2015, having the terms "fatigue," "drowsiness," "vigilance," or "alertness" in the title, and proposing wearable device-based systems for non-invasive fatigue quantification. Of the 612 retrieved articles, 60 satisfied the inclusion criteria. Included studies were mainly of short duration and conducted in laboratory settings. In general, researchers developed fatigue models based on motion (MOT), electroencephalogram (EEG), photoplethysmogram (PPG), electrocardiogram (ECG), galvanic skin response (GSR), electromyogram (EMG), skin temperature (T_sk), eye movement (EYE), and respiratory (RES) data acquired by wearable devices available in the market. Supervised machine learning models, and more specifically, binary classification models, are predominant among the proposed fatigue quantification approaches. These models were considered to perform very well in detecting fatigue, however, little effort was made to ensure the use of high-quality data during model development. Together, the findings of this review reveal that methodological limitations have hindered the generalizability and real-world applicability of most of the proposed fatigue models. Considerably more work is needed to fully explore the potential of wearables for fatigue quantification as well as to better understand the relationship between fatigue and changes in physiological variables.

The Impact of Control Interface on Features of Heart Rate Variability

Shared human-robot control for assistive machines can improve the independence of individuals with motor impairments. Monitoring elevated levels of workload can enable the assistive autonomy to adjust the control sharing in an assist-as-needed way, to achieve a balance between user fatigue, stress, and independent control. In this work, we aim to investigate how heart rate variability features can be utilized to monitor elevated levels of mental workload while operating a powered wheelchair, and how that utilization might vary under different control interfaces. To that end, we conduct a 22 person study with three commercial interfaces. Our results show that the validity and reliability of using the ultra-short-term heart-rate variability features as predictors of workload indeed are affected by the type of interface in use.

SSVEP-EEG Denoising via Image Filtering Methods

Steady-state visual evoked potential (SSVEP) is widely used in electroencephalogram (EEG) control, medical detection, cognitive neuroscience, and other fields. However, successful application requires improving the detection performance of SSVEP signal frequency characteristics. Most strategies to enhance the signal-to-noise ratio of SSVEP utilize application of a spatial filter. Here, we propose a method for image filtering denoising (IFD) of the SSVEP signal from the perspective of image analysis, as a preprocessing step for signal analysis. Arithmetic mean, geometric mean, Gaussian, and non-local means filtering methods were tested, and the experimental results show that image filtering of SSVEP cannot effectively remove the noise. Thus, we proposed a reverse solution in which the SSVEP noise signal was obtained by image filtering, and then the noise was subtracted from the original signal. Comparison of the recognition accuracy of the SSVEP signal before and after denoising was used to evaluate the denoising performance for stimuli of different duration. After IFD processing, SSVEP exhibited higher recognition accuracy, indicating the effectiveness of this proposed denoising method. Application of this method improves the detection performance of SSVEP signal frequency characteristics, combines image processing and brain signal analysis, and expands the research scope of brain signal analysis for widespread application.

A Muscular and Cerebral Physiological Indices Assessment for Stress Measuring during Virtual Wheelchair Guidance

The work presented in this manuscript has the purpose to assess the relationship between human factors and physiological indices. We discuss the relationship between stress as human factor and cerebral and muscular signals as features. Ten male paraplegic, right-handed subjects were volunteers for the experiment (mean age 34 ±6). They drove a virtual wheelchair in an indoor environment. They filled five missions where, in each one, an environmental parameter was changed. Meanwhile, they were equipped with Electromyography (EMG) sensors and Electroencephalography (EEG). Frequency and temporal features were filtered and extracted. Principal component analysis (PCA), Fisher's tests, repeated measure Anova and post hoc Tukey test (α = 0.05) were implemented for statistics. Environmental modifications are subject to induce stress, which impacts muscular and cerebral activities. While the time pressure parameter was the most influent, the transition from static to moving obstacles (avatars), tends to have a significant impact on stress levels. However, adding more moving obstacles did not show any impact. A synchronization factor was noticed between cerebral and muscular features in higher stress levels. Further examination is needed to assess EEG reliability in these situations.

BCI-Based Rehabilitation on the Stroke in Sequela Stage

Background

Stroke is the leading cause of serious and long-term disability worldwide. Survivors may recover some motor functions after rehabilitation therapy. However, many stroke patients missed the best time period for recovery and entered into the sequela stage of chronic stroke.

Method

Studies have shown that motor imagery- (MI-) based brain-computer interface (BCI) has a positive effect on poststroke rehabilitation. This study used both virtual limbs and functional electrical stimulation (FES) as feedback to provide patients with a closed-loop sensorimotor integration for motor rehabilitation. An MI-based BCI system acquired, analyzed, and classified motor attempts from electroencephalogram (EEG) signals. The FES system would be activated if the BCI detected that the user was imagining wrist dorsiflexion on the instructed side of the body. Sixteen stroke patients in the sequela stage were randomly assigned to a BCI group and a control group. All of them participated in rehabilitation training for four weeks and were assessed by the Fugl-Meyer Assessment (FMA) of motor function.

Results

The average improvement score of the BCI group was 3.5, which was higher than that of the control group (0.9). The active EEG patterns of the four patients in the BCI group whose FMA scores increased gradually became centralized and shifted to sensorimotor areas and premotor areas throughout the study.

Conclusions

Study results showed evidence that patients in the BCI group achieved larger functional improvements than those in the control group and that the BCI-FES system is effective in restoring motor function to upper extremities in stroke patients. This study provides a more autonomous approach than traditional treatments used in stroke rehabilitation.

Instantaneous mental workload assessment using time-frequency analysis and semi-supervised learning

The real-time assessment of mental workload (MWL) is critical for development of intelligent human–machine cooperative systems in various safety–critical applications. Although data-driven machine learning (ML) approach has shown promise in MWL recognition, there is still difficulty in acquiring a sufficient number of labeled data to train the ML models. This paper proposes a semi-supervised extreme learning machine (SS-ELM) algorithm for MWL pattern classification requiring only a small number of labeled data. The measured data analysis results show that the proposed SS-ELM paradigm can effectively improve the accuracy and efficiency of MWL classification and thus provide a competitive ML approach to utilizing a large number of unlabeled data which are available in many real-world applications.

Brain-computer interface method based on light-flashing and motion hybrid coding

The human best response frequency band for steady-state visual evoked potential stimulus is limited. This results in a reduced number of encoded targets. To circumvent this, we proposed a brain-computer interface (BCI) method based on light-flashing and motion hybrid coding. The hybrid paradigm pattern consisted of a circular light-flashing pattern and a motion pattern located in the inner ring of light-flashing pattern. The motion and light-flashing patterns had different frequencies. This study used five frequencies to encode nine targets. The motion frequency and the light-flashing frequency of the hybrid paradigm consisted of two frequencies in five frequencies. The experimental results showed that the hybrid paradigm could induce stable motion frequency, light-flashing frequency and its harmonic components. Moreover, the modulation between motion and light-flashing was weak. The average accuracy was 92.96% and the information transfer rate was 26.10 bits/min. The experimental results showed that the proposed method could be considered for practical BCI systems.

Neural mechanism of visual information degradation from retina to V1 area

The information processing mechanism of the visual nervous system is an unresolved scientific problem that has long puzzled neuroscientists. The amount of visual information is significantly degraded when it reaches the V1 after entering the retina; nevertheless, this does not affect our visual perception of the outside world. Currently, the mechanisms of visual information degradation from retina to V1 are still unclear. For this purpose, the current study used the experimental data summarized by Marcus E. Raichle to investigate the neural mechanisms underlying the degradation of the large amount of data from topological mapping from retina to V1, drawing on the photoreceptor model first. The obtained results showed that the image edge features of visual information were extracted by the convolution algorithm with respect to the function of synaptic plasticity when visual signals were hierarchically processed from low-level to high-level. The visual processing was characterized by the visual information degradation, and this compensatory mechanism embodied the principles of energy minimization and transmission efficiency maximization of brain activity, which matched the experimental data summarized by Marcus E. Raichle. Our results further the understanding of the information processing mechanism of the visual nervous system.

Medium- and long-term functional behavior evaluations in an experimental focal ischemic stroke mouse model

Cerebrovascular accident (CVA) is one of the leading causes of death and disability worldwide, as well as a major financial burden for health care systems. CVA rodent models provide experimental support to determine possible in vivo therapies to reduce brain injury and consequent sequelae. This study analyzed nociceptive, motor, cognitive and mood functions in mice submitted to distal middle cerebral artery (DMCA) occlusion. Male C57BL mice (n = 8) were randomly allocated to control or DMCA groups. Motor function was evaluated with the tests: grip force, rotarod and open field; and nociceptive threshold with von Frey and hot plate assessments. Cognitive function was evaluated with the inhibitory avoidance test, and mood with the tail suspension test. Evaluations were conducted on the seventh- and twenty-eighth-day post DMCA occlusion to assess medium- and long-term effects of the injury, respectively. DMCA occlusion significantly decreases muscle strength and spontaneous locomotion (p < 0.05) both medium- and long term; as well as increases immobility in the tail-suspension test (p < 0.05), suggesting a depressive-type behavior. However, DMCA occlusion did not affect nociceptive threshold nor cognitive functions (p > 0.05). These results suggest that, medium- and long-term effects of DMCA occlusion include motor function impairments, but no sensory dysfunction. Additionally, the injury affected mood but did not hinder cognitive function.

Adaptive Neuro-Fuzzy Fusion of Multi-Sensor Data for Monitoring a Pilot's Workload Condition

To realize an early warning of unbalanced workload in the aircraft cockpit, it is required to monitor the pilot’s real-time workload condition. For the purpose of building the mapping relationship from physiological and flight data to workload, a multi-source data fusion model is proposed based on a fuzzy neural network, mainly structured using a principal components extraction layer, fuzzification layer, fuzzy rules matching layer, and normalization layer. Aiming at the high coupling characteristic variables contributing to workload, principal component analysis reconstructs the feature data by reducing its dimension. Considering the uncertainty for a single variable to reflect overall workload, a fuzzy membership function and fuzzy control rules are defined to abstract the inference process. An error feedforward algorithm based on gradient descent is utilized for parameter learning. Convergence speed and accuracy can be adjusted by controlling the gradient descent rate and error tolerance threshold. Combined with takeoff and initial climbing tasks of a Boeing 737–800 aircraft, crucial performance indicators—including pitch angle, heading, and airspeed—as well as physiological indicators—including electrocardiogram (ECG), respiration, and eye movements—were featured. The mapping relationship between multi-source data and the comprehensive workload level synthesized using the NASA task load index was established. Experimental results revealed that the predicted workload corresponding to different flight phases and difficulty levels showed clear distinctions, thereby proving the validity of data fusion.