Kinesthesia in a sustained-attention driving task

DTP-Net: Learning to Reconstruct EEG Signals in Time-Frequency Domain by Multi-Scale Feature Reuse

Electroencephalography (EEG) signals are prone to contamination by noise, such as ocular and muscle artifacts. Minimizing these artifacts is crucial for EEG-based downstream applications like disease diagnosis and brain-computer interface (BCI). This paper presents a new EEG denoising model, DTP-Net. It is a fully convolutional neural network comprising Densely-connected Temporal Pyramids (DTPs) placed between two learnable time-frequency transformations. In the time-frequency domain, DTPs facilitate efficient propagation of multi-scale features extracted from EEG signals of any length, leading to effective noise reduction. Comprehensive experiments on two public semi-simulated datasets demonstrate that the proposed DTP-Net consistently outperforms existing state-of-the-art methods on metrics including relative root mean square error (RRMSE) and signal-to-noise ratio improvement ( ∆SNR). Moreover, the proposed DTP-Net is applied to a BCI classification task, yielding an improvement of up to 5.55% in accuracy. This confirms the potential of DTP-Net for applications in the fields of EEG-based neuroscience and neuro-engineering. An in-depth analysis further illustrates the representation learning behavior of each module in DTP-Net, demonstrating its robustness and reliability.

IC-U-Net: A U-Net-based Denoising Autoencoder Using Mixtures of Independent Components for Automatic EEG Artifact Removal

Electroencephalography (EEG) signals are often contaminated with artifacts. It is imperative to develop a practical and reliable artifact removal method to prevent the misinterpretation of neural signals and the underperformance of brain-computer interfaces. Based on the U-Net architecture, we developed a new artifact removal model, IC-U-Net, for removing pervasive EEG artifacts and reconstructing brain signals. IC-U-Net was trained using mixtures of brain and non-brain components decomposed by independent component analysis. It uses an ensemble of loss functions to model complex signal fluctuations in EEG recordings. The effectiveness of the proposed method in recovering brain activities and removing various artifacts (e.g., eye blinks/movements, muscle activities, and line/channel noise) was demonstrated in a simulation study and four real-world EEG experiments. IC-U-Net can reconstruct a multi-channel EEG signal and is applicable to most artifact types, offering a promising end-to-end solution for automatically removing artifacts from EEG recordings. It also meets the increasing need to image natural brain dynamics in a mobile setting. The code and pre-trained IC-U-Net model are available at https://github.com/roseDwayane/AIEEG.

The impact of image degradation and temporal dynamics on sustained attention

Many clinical populations that have sustained attention deficits also have visual deficits. Therefore, it is necessary to understand how the quality of visual input and different forms of image degradation can contribute to worse performance on sustained attention tasks, particularly those with dynamic and complex visual stimuli. This study investigated the impact of image degradation on an adapted version of the gradual-onset continuous performance task (gradCPT), where participants must discriminate between gradually fading city and mountain scenes. Thirty-six normal-vision participants completed the task, which featured two blocks of six resolution and contrast levels. Subjects either completed a version with gradually fading or static image presentations. The results show decreases in image resolution impair performance under both types of temporal dynamics, whereas performance is only impaired under gradual temporal dynamics for decreases in image contrast. Image similarity analyses showed that performance has a higher association with an observer's ability to gather an image's global spatial layout (i.e. gist) than local variations in pixel luminance, particularly under gradual image presentation. This work suggests that gradually fading attention paradigms are sensitive to deficits in primary visual function, potentially leading to these issues being misinterpreted as attentional failures.

Spatiotemporal Characteristics of Event-Related Potentials Triggered by Unexpected Events during Simulated Driving and Influence of Vigilance

We investigated the spatiotemporal characteristics of brain activity due to sudden events during monotonous driving and how it changes with vigilance level. Two types of sudden events, emergency stop and car drifting, were presented using driving simulator, and event-related potentials (ERPs) were measured. From the ERPs of both types of events, an early component representing sensory information processing and a late component were observed. The early component was expected to represent sensory information processing, which corresponded to visual and somatosensory/vestibular information processing for the sudden stop and lane departure tasks, respectively. The late components showed spatiotemporal characteristics of the well-known P300 component for both types of events. Common characteristic brain activities occurred in response to sudden events, regardless of the type. The modulation of brain activity due to the vigilance level also shared common characteristics between the two types. We expect that our results will contribute to the development of an effective means to assist drivers’ reactions to ambulatory situations.

EEG Correlates of Sustained Attention Variability during Discrete Multi-finger Force Control Tasks

The neurophysiological characteristics of sustained attention states are unclear in discrete multi-finger force control tasks. In this article, we developed an immersive visuo-haptic task for conducting stimulus-response measurements. Visual cues were randomly provided to signify the required amplitude and tolerance of fingertip force. Participants were required to respond to the visual cues by pressing force transducers using their fingertips. Response time variation was taken as a behavioral measure of sustained attention states during the task. 50% low-variability trials were classified as the optimal state and the other high-variability trials were classified as the suboptimal state using z-scoring over time. A 64-channel electroencephalogram (EEG) acquisition system was used to collect brain activities during the tasks. The haptics-elicited potential amplitude at 20 ∼ 40 ms in latency and over the frontal-central region significantly decreased in the optimal state. Furthermore, the alpha-band power in the spectra of 8 ∼ 13 Hz was significantly suppressed in the frontal-central, right temporal, and parietal regions in the optimal state. Taken together, we have identified neuroelectrophysiological features that were associated with sustained attention during multi-finger force control tasks, which would be potentially used in the development of closed-loop attention detection and training systems exploiting haptic interaction.

Exploring the Brain Responses to Driving Fatigue Through Simultaneous EEG and fNIRS Measurements

Fatigue is one problem with driving as it can lead to difficulties with sustaining attention, behavioral lapses, and a tendency to ignore vital information or operations. In this research, we explore multimodal physiological phenomena in response to driving fatigue through simultaneous functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) recordings with the aim of investigating the relationships between hemodynamic and electrical features and driving performance. Sixteen subjects participated in an event-related lane-deviation driving task while measuring their brain dynamics through fNIRS and EEGs. Three performance groups, classified as Optimal, Suboptimal, and Poor, were defined for comparison. From our analysis, we find that tonic variations occur before a deviation, and phasic variations occur afterward. The tonic results show an increased concentration of oxygenated hemoglobin (HbO₂) and power changes in the EEG theta, alpha, and beta bands. Both dynamics are significantly correlated with deteriorated driving performance. The phasic EEG results demonstrate event-related desynchronization associated with the onset of steering vehicle in all power bands. The concentration of phasic HbO₂ decreased as performance worsened. Further, the negative correlations between tonic EEG delta and alpha power and HbO₂ oscillations suggest that activations in HbO₂ are related to mental fatigue. In summary, combined hemodynamic and electrodynamic activities can provide complete knowledge of the brain's responses as evidence of state changes during fatigue driving.

Electroencephalographic and peripheral temperature dynamics during a prolonged psychomotor vigilance task

Attention lapses and fatigue are a main source of impaired performance that can lead to accidents. This study analyzed electroencephalographic (EEG) dynamics and body skin temperature as markers of attentional fluctuations in non-sleep deprived subjects during a 45min Psychomotor Vigilance Task (PVT). Independent Component Analysis and time-frequency analysis were used to evaluate the EEG data. Results showed a positive association between distal and distal-to-proximal gradient (DPG) temperatures and reaction time (RT); increments in EEG power in alpha-, theta- and beta-band frequencies in parieto-occipital, central-medial and frontal components, were associated with poor performance (slower RT) in the task. This generalized power increment fits with an increased activity in the default mode network, associated with attention lapses. This study highlights the potential use of the PVT as a tool to obtain individual physiological indices of vigilance and fatigue that could be applied to other vigilance tasks typically performed in occupational settings.

Multi-channel EEG recordings during a sustained-attention driving task

We describe driver behaviour and brain dynamics acquired from a 90-minute sustained-attention task in an immersive driving simulator. The data included 62 sessions of 32-channel electroencephalography (EEG) data for 27 subjects driving on a four-lane highway who were instructed to keep the car cruising in the centre of the lane. Lane-departure events were randomly induced to cause the car to drift from the original cruising lane towards the left or right lane. A complete trial included events with deviation onset, response onset, and response offset. The next trial, in which the subject was instructed to drive back to the original cruising lane, began 5-10 seconds after finishing the previous trial. We believe that this dataset will lead to the development of novel neural processing methodology that can be used to index brain cortical dynamics and detect driving fatigue and drowsiness. This publicly available dataset will be beneficial to the neuroscience and brain-computer interface communities.

Brain Network Changes in Fatigued Drivers: A Longitudinal Study in a Real-World Environment Based on the Effective Connectivity Analysis and Actigraphy Data

The analysis of neurophysiological changes during driving can clarify the mechanisms of fatigue, considered an important cause of vehicle accidents. The fluctuations in alertness can be investigated as changes in the brain network connections, reflected in the direction and magnitude of the information transferred. Those changes are induced not only by the time on task but also by the quality of sleep. In an unprecedented 5-month longitudinal study, daily sampling actigraphy and EEG data were collected during a sustained-attention driving task within a near-real-world environment. Using a performance index associated with the subjects' reaction times and a predictive score related to the sleep quality, we identify fatigue levels in drivers and investigate the shifts in their effective connectivity in different frequency bands, through the analysis of the dynamical coupling between brain areas. Study results support the hypothesis that combining EEG, behavioral and actigraphy data can reveal new features of the decline in alertness. In addition, the use of directed measures such as the Convergent Cross Mapping can contribute to the development of fatigue countermeasure devices.

An analysis on driver drowsiness based on reaction time and EEG band power

Falling asleep during driving is a serious problem that has resulted in fatal accidents worldwide. Thus, there is a need to detect driver drowsiness to counter it. This study analyzes the changes in the electroencephalography (EEG) collected from 4 subjects driving under monotonous road conditions using a driving simulator. The drowsiness level of the subjects is inferred from the time taken to react to events. The results from the analysis of the reaction time shows that drowsiness occurs in cycles, which correspond to short sleep cycles known as `microsleeps'. The results from a time-frequency analysis of the four frequency bands' power reveals differences between trials with fast and slow reaction times; greater beta band power is present in all subjects, greater alpha power in 2 subjects, greater theta power in 2 subjects, and greater delta power in 3 subjects, for fast reaction trials. Overall, this study shows that reaction time can be used to infer the drowsiness, and subject-specific changes in the EEG band power may be used to infer drowsiness. Thus the study shows a promising prospect of developing Brain-Computer Interface to detect driver drowsiness.

Brain Electrodynamic and Hemodynamic Signatures Against Fatigue During Driving

Fatigue is likely to be gradually cumulated in a prolonged and attention-demanding task that may adversely affect task performance. To address the brain dynamics during a driving task, this study recruited 16 subjects to participate in an event-related lane-departure driving experiment. Each subject was instructed to maintain attention and task performance throughout an hour-long driving experiment. The subjects' brain electrodynamics and hemodynamics were simultaneously recorded via 32-channel electroencephalography (EEG) and 8-source/16-detector functional near-infrared spectroscopy (fNIRS). The behavior performance demonstrated that all subjects were able to promptly respond to lane-deviation events, even if the sign of fatigue arose in the brain, which suggests that the subjects were fighting fatigue during the driving experiment. The EEG event-related analysis showed strengthening alpha suppression in the occipital cortex, a common brain region of fatigue. Furthermore, we noted increasing oxygenated hemoglobin (HbO) of the brain to fight driving fatigue in the frontal cortex, primary motor cortex, parieto-occipital cortex and supplementary motor area. In conclusion, the increasing neural activity and cortical activations were aimed at maintaining driving performance when fatigue emerged. The electrodynamic and hemodynamic signatures of fatigue fighting contribute to our understanding of the brain dynamics of driving fatigue and address driving safety issues through the maintenance of attention and behavioral performance.

A subject-transfer framework for obviating inter- and intra-subject variability in EEG-based drowsiness detection

Inter- and intra-subject variability pose a major challenge to decoding human brain activity in brain-computer interfaces (BCIs) based on non-invasive electroencephalogram (EEG). Conventionally, a time-consuming and laborious training procedure is performed on each new user to collect sufficient individualized data, hindering the applications of BCIs on monitoring brain states (e.g. drowsiness) in real-world settings. This study proposes applying hierarchical clustering to assess the inter- and intra-subject variability within a large-scale dataset of EEG collected in a simulated driving task, and validates the feasibility of transferring EEG-based drowsiness-detection models across subjects. A subject-transfer framework is thus developed for detecting drowsiness based on a large-scale model pool from other subjects and a small amount of alert baseline calibration data from a new user. The model pool ensures the availability of positive model transferring, whereas the alert baseline data serve as a selector of decoding models in the pool. Compared with the conventional within-subject approach, the proposed framework remarkably reduced the required calibration time for a new user by 90% (18.00 min-1.72 ± 0.36 min) without compromising performance (p = 0.0910) when sufficient existing data are available. These findings suggest a practical pathway toward plug-and-play drowsiness detection and can ignite numerous real-world BCI applications.

Time-Frequency Analysis of Mu Rhythm Activity during Picture and Video Action Naming Tasks

This study used whole-head 64 channel electroencephalography to measure changes in sensorimotor activity—as indexed by the mu rhythm—in neurologically-healthy adults, during subvocal confrontation naming tasks. Independent component analyses revealed sensorimotor mu component clusters in the right and left hemispheres. Event related spectral perturbation analyses indicated significantly stronger patterns of mu rhythm activity (pFDR < 0.05) during the video condition as compared to the picture condition, specifically in the left hemisphere. Mu activity is hypothesized to reflect typical patterns of sensorimotor activation during action verb naming tasks. These results support further investigation into sensorimotor cortical activity during action verb naming in clinical populations.

BLINKER: Automated Extraction of Ocular Indices from EEG Enabling Large-Scale Analysis

Electroencephalography (EEG) offers a platform for studying the relationships between behavioral measures, such as blink rate and duration, with neural correlates of fatigue and attention, such as theta and alpha band power. Further, the existence of EEG studies covering a variety of subjects and tasks provides opportunities for the community to better characterize variability of these measures across tasks and subjects. We have implemented an automated pipeline (BLINKER) for extracting ocular indices such as blink rate, blink duration, and blink velocity-amplitude ratios from EEG channels, EOG channels, and/or independent components (ICs). To illustrate the use of our approach, we have applied the pipeline to a large corpus of EEG data (comprising more than 2000 datasets acquired at eight different laboratories) in order to characterize variability of certain ocular indicators across subjects. We also investigate dependence of ocular indices on task in a shooter study. We have implemented our algorithms in a freely available MATLAB toolbox called BLINKER. The toolbox, which is easy to use and can be applied to collections of data without user intervention, can automatically discover which channels or ICs capture blinks. The tools extract blinks, calculate common ocular indices, generate a report for each dataset, dump labeled images of the individual blinks, and provide summary statistics across collections. Users can run BLINKER as a script or as a plugin for EEGLAB. The toolbox is available at https://github.com/VisLab/EEG-Blinks. User documentation and examples appear at http://vislab.github.io/EEG-Blinks/.

Hierarchical Event Descriptors (HED): Semi-Structured Tagging for Real-World Events in Large-Scale EEG

Real-world brain imaging by EEG requires accurate annotation of complex subject-environment interactions in event-rich tasks and paradigms. This paper describes the evolution of the Hierarchical Event Descriptor (HED) system for systematically describing both laboratory and real-world events. HED version 2, first described here, provides the semantic capability of describing a variety of subject and environmental states. HED descriptions can include stimulus presentation events on screen or in virtual worlds, experimental or spontaneous events occurring in the real world environment, and events experienced via one or multiple sensory modalities. Furthermore, HED 2 can distinguish between the mere presence of an object and its actual (or putative) perception by a subject. Although the HED framework has implicit ontological and linked data representations, the user-interface for HED annotation is more intuitive than traditional ontological annotation. We believe that hiding the formal representations allows for a more user-friendly interface, making consistent, detailed tagging of experimental, and real-world events possible for research users. HED is extensible while retaining the advantages of having an enforced common core vocabulary. We have developed a collection of tools to support HED tag assignment and validation; these are available at hedtags.org. A plug-in for EEGLAB (sccn.ucsd.edu/eeglab), CTAGGER, is also available to speed the process of tagging existing studies.

EEG and Eye Tracking Demonstrate Vigilance Enhancement with Challenge Integration

Maintaining vigilance is possibly the first requirement for surveillance tasks where personnel are faced with monotonous yet intensive monitoring tasks. Decrement in vigilance in such situations could result in dangerous consequences such as accidents, loss of life and system failure. In this paper, we investigate the possibility to enhance vigilance or sustained attention using “challenge integration,” a strategy that integrates a primary task with challenging stimuli. A primary surveillance task (identifying an intruder in a simulated factory environment) and a challenge stimulus (periods of rain obscuring the surveillance scene) were employed to test the changes in vigilance levels. The effect of integrating challenging events (resulting from artificially simulated rain) into the task were compared to the initial monotonous phase. EEG and eye tracking data is collected and analyzed for n = 12 subjects. Frontal midline theta power and frontal theta to parietal alpha power ratio which are used as measures of engagement and attention allocation show an increase due to challenge integration (p < 0.05 in each case). Relative delta band power of EEG also shows statistically significant suppression on the frontoparietal and occipital cortices due to challenge integration (p < 0.05). Saccade amplitude, saccade velocity and blink rate obtained from eye tracking data exhibit statistically significant changes during the challenge phase of the experiment (p < 0.05 in each case). From the correlation analysis between the statistically significant measures of eye tracking and EEG, we infer that saccade amplitude and saccade velocity decrease with vigilance decrement along with frontal midline theta and frontal theta to parietal alpha ratio. Conversely, blink rate and relative delta power increase with vigilance decrement. However, these measures exhibit a reverse trend when challenge stimulus appears in the task suggesting vigilance enhancement. Moreover, the mean reaction time is lower for the challenge integrated phase (RTmean = 3.65 ± 1.4s) compared to initial monotonous phase without challenge (RTmean = 4.6 ± 2.7s). Our work shows that vigilance level, as assessed by response of these vital signs, is enhanced by challenge integration.

Mind-Wandering Tends to Occur under Low Perceptual Demands during Driving

Fluctuations in attention behind the wheel poses a significant risk for driver safety. During transient periods of inattention, drivers may shift their attention towards internally-directed thoughts or feelings at the expense of staying focused on the road. This study examined whether increasing task difficulty by manipulating involved sensory modalities as the driver detected the lane-departure in a simulated driving task would promote a shift of brain activity between different modes of processing, reflected by brain network dynamics on electroencephalographic sources. Results showed that depriving the driver of salient sensory information imposes a relatively more perceptually-demanding task, leading to a stronger activation in the task-positive network. When the vehicle motion feedback is available, the drivers may rely on vehicle motion to perceive the perturbations, which frees attentional capacity and tends to activate the default mode network. Such brain network dynamics could have major implications for understanding fluctuations in driver attention and designing advance driver assistance systems.

Brain Dynamics in Predicting Driving Fatigue Using a Recurrent Self-Evolving Fuzzy Neural Network

This paper proposes a generalized prediction system called a recurrent self-evolving fuzzy neural network (RSEFNN) that employs an on-line gradient descent learning rule to address the electroencephalography (EEG) regression problem in brain dynamics for driving fatigue. The cognitive states of drivers significantly affect driving safety; in particular, fatigue driving, or drowsy driving, endangers both the individual and the public. For this reason, the development of brain-computer interfaces (BCIs) that can identify drowsy driving states is a crucial and urgent topic of study. Many EEG-based BCIs have been developed as artificial auxiliary systems for use in various practical applications because of the benefits of measuring EEG signals. In the literature, the efficacy of EEG-based BCIs in recognition tasks has been limited by low resolutions. The system proposed in this paper represents the first attempt to use the recurrent fuzzy neural network (RFNN) architecture to increase adaptability in realistic EEG applications to overcome this bottleneck. This paper further analyzes brain dynamics in a simulated car driving task in a virtual-reality environment. The proposed RSEFNN model is evaluated using the generalized cross-subject approach, and the results indicate that the RSEFNN is superior to competing models regardless of the use of recurrent or nonrecurrent structures.

Identifying changes in EEG information transfer during drowsy driving by transfer entropy

Drowsy driving is a major cause of automobile accidents. Previous studies used neuroimaging based approaches such as analysis of electroencephalogram (EEG) activities to understand the brain dynamics of different cortical regions during drowsy driving. However, the coupling between brain regions responding to this vigilance change is still unclear. To have a comprehensive understanding of neural mechanisms underlying drowsy driving, in this study we use transfer entropy, a model-free measure of effective connectivity based on information theory. We investigate the pattern of information transfer between brain regions when the vigilance level, which is derived from the driving performance, changes from alertness to drowsiness. Results show that the couplings between pairs of frontal, central, and parietal areas increased at the intermediate level of vigilance, which suggests that an enhancement of the cortico-cortical interaction is necessary to maintain the task performance and prevent behavioral lapses. Additionally, the occipital-related connectivity magnitudes monotonically decreases as the vigilance level declines, which further supports the cortical gating of sensory stimuli during drowsiness. Neurophysiological evidence of mutual relationships between brain regions measured by transfer entropy might enhance the understanding of cortico-cortical communication during drowsy driving.

The PREP pipeline: standardized preprocessing for large-scale EEG analysis

The technology to collect brain imaging and physiological measures has become portable and ubiquitous, opening the possibility of large-scale analysis of real-world human imaging. By its nature, such data is large and complex, making automated processing essential. This paper shows how lack of attention to the very early stages of an EEG preprocessing pipeline can reduce the signal-to-noise ratio and introduce unwanted artifacts into the data, particularly for computations done in single precision. We demonstrate that ordinary average referencing improves the signal-to-noise ratio, but that noisy channels can contaminate the results. We also show that identification of noisy channels depends on the reference and examine the complex interaction of filtering, noisy channel identification, and referencing. We introduce a multi-stage robust referencing scheme to deal with the noisy channel-reference interaction. We propose a standardized early-stage EEG processing pipeline (PREP) and discuss the application of the pipeline to more than 600 EEG datasets. The pipeline includes an automatically generated report for each dataset processed. Users can download the PREP pipeline as a freely available MATLAB library from http://eegstudy.org/prepcode.