Physiological characterization of electrodermal activity enables scalable near real-time autonomic nervous system activation inference

A multimodal dataset for investigating working memory in presence of music: a pilot study

Introduction

Decoding an individual's hidden brain states in responses to musical stimuli under various cognitive loads can unleash the potential of developing a non-invasive closed-loop brain-machine interface (CLBMI). To perform a pilot study and investigate the brain response in the context of CLBMI, we collect multimodal physiological signals and behavioral data within the working memory experiment in the presence of personalized musical stimuli.

Methods

Participants perform a working memory experiment called the n-back task in the presence of calming music and exciting music. Utilizing the skin conductance signal and behavioral data, we decode the brain's cognitive arousal and performance states, respectively. We determine the association of oxygenated hemoglobin (HbO) data with performance state. Furthermore, we evaluate the total hemoglobin (HbT) signal energy over each music session.

Results

A relatively low arousal variation was observed with respect to task difficulty, while the arousal baseline changes considerably with respect to the type of music. Overall, the performance index is enhanced within the exciting session. The highest positive correlation between the HbO concentration and performance was observed within the higher cognitive loads (3-back task) for all of the participants. Also, the HbT signal energy peak occurs within the exciting session.

Discussion

Findings may underline the potential of using music as an intervention to regulate the brain cognitive states. Additionally, the experiment provides a diverse array of data encompassing multiple physiological signals that can be used in the brain state decoder paradigm to shed light on the human-in-the-loop experiments and understand the network-level mechanisms of auditory stimulation.

Non-invasive detection of mental fatigue in construction equipment operators through geometric measurements of facial features

INTRODUCTION

Prolonged operation of construction equipment could lead to mental fatigue, which can increase the chances of human error-related accidents as well as operators' ill-health. The objective detection of operators' mental fatigue is crucial for reducing accident risk and ensuring operator health. Electroencephalography, photoplethysmography, electrodermal activity, and eye-tracking technology have been used to mitigate this issue. These technologies are invasive and wearable sensors that can cause irritation and discomfort. Geometric measurements of facial features can serve as a noninvasive alternative approach. Its application in detecting mental fatigue of construction equipment operators has not been reported in the literature. Although the application of facial features has been widespread in other domains, such as drivers and other occupation scenarios, their ecological validity for construction excavator operators remains a knowledge gap.

METHOD

This study proposed employing geometric measurements of facial features to detect mental fatigue in construction equipment operators' facial features. In this study, seventeen operators performed excavation operations. Mental fatigue was labeled subjectively and objectively using NASA-TLX scores and EDA values. Based on geometric measurements, facial features (eyebrow, mouth outer, mouth corners, head motion, eye area, and face area) were extracted.

RESULTS

The results showed that there was significant difference in the measured metrics for high fatigue compared to low fatigue. Specifically, the most noteworthy variation was for the eye and face area metrics, with mean differences of 45.88% and 26.9%, respectively.

CONCLUSIONS

The findings showed that geometrical measurements of facial features are a useful, noninvasive approach for detecting the mental fatigue of construction equipment operators.

Unveiling productivity: The interplay of cognitive arousal and expressive typing in remote work

Cognitive Arousal, frequently elicited by environmental stressors that exceed personal coping resources, manifests in measurable physiological markers, notably in galvanic skin responses. This effect is prominent in cognitive tasks such as composition, where fluctuations in these biomarkers correlate with individual expressiveness. It is crucial to understand the nexus between cognitive arousal and expressiveness. However, there has not been a concrete study that investigates this inter-relation concurrently. Addressing this, we introduce an innovative methodology for simultaneous monitoring of these elements. Our strategy employs Bayesian analysis in a multi-state filtering format to dissect psychomotor performance (captured through typing speed), galvanic skin response or skin conductance (SC), and heart rate variability (HRV). This integrative analysis facilitates the quantification of expressive behavior and arousal states. At the core, we deploy a state-space model connecting one latent psychological arousal condition to neural activities impacting sweating (inferred through SC responses) and another latent state to expressive behavior during typing. These states are concurrently evaluated with model parameters using an expectation-maximization algorithms approach. Assessments using both computer-simulated data and experimental data substantiate the validity of our approach. Outcomes display distinguishable latent state patterns in expressive typing and arousal across different computer software used in office management, offering profound implications for Human-Computer Interaction (HCI) and productivity analysis. This research marks a significant advancement in decoding human productivity dynamics, with extensive repercussions for optimizing performance in telecommuting scenarios.

Design of a Remote Multiparametric Tool to Assess Mental Well-Being and Distress in Young People (mHealth Methods in Mental Health Research Project): Protocol for an Observational Study

BACKGROUND

Mental health conditions have become a substantial cause of disability worldwide, resulting in economic burden and strain on the public health system. Incorporating cognitive and physiological biomarkers using noninvasive sensors combined with self-reported questionnaires can provide a more accurate characterization of the individual's well-being. Biomarkers such as heart rate variability or those extracted from the electrodermal activity signal are commonly considered as indices of autonomic nervous system functioning, providing objective indicators of stress response. A model combining a set of these biomarkers can constitute a comprehensive tool to remotely assess mental well-being and distress.

OBJECTIVE

This study aims to design and validate a remote multiparametric tool, including physiological and cognitive variables, to objectively assess mental well-being and distress.

METHODS

This ongoing observational study pursues to enroll 60 young participants (aged 18-34 years) in 3 groups, including participants with high mental well-being, participants with mild to moderate psychological distress, and participants diagnosed with depression or anxiety disorder. The inclusion and exclusion criteria are being evaluated through a web-based questionnaire, and for those with a mental health condition, the criteria are identified by psychologists. The assessment consists of collecting mental health self-reported measures and physiological data during a baseline state, the Stroop Color and Word Test as a stress-inducing stage, and a final recovery period. Several variables related to heart rate variability, pulse arrival time, breathing, electrodermal activity, and peripheral temperature are collected using medical and wearable devices. A second assessment is carried out after 1 month. The assessment tool will be developed using self-reported questionnaires assessing well-being (short version of Warwick-Edinburgh Mental Well-being Scale), anxiety (Generalized Anxiety Disorder-7), and depression (Patient Health Questionnaire-9) as the reference. We will perform correlation and principal component analysis to reduce the number of variables, followed by the calculation of multiple regression models. Test-retest reliability, known-group validity, and predictive validity will be assessed.

RESULTS

Participant recruitment is being carried out on a university campus and in mental health services. Recruitment commenced in October 2022 and is expected to be completed by June 2024. As of July 2023, we have recruited 41 participants. Most participants correspond to the group with mild to moderate psychological distress (n=20, 49%), followed by the high mental well-being group (n=13, 32%) and those diagnosed with a mental health condition (n=8, 20%). Data preprocessing is currently ongoing, and publication of the first results is expected by September 2024.

CONCLUSIONS

This study will establish an initial framework for a comprehensive mental health assessment tool, taking measurements from sophisticated devices, with the goal of progressing toward a remotely accessible and objectively measured approach that maintains an acceptable level of accuracy in clinical practice and epidemiological studies.

TRIAL REGISTRATION

OSF Registries N3GCH; https://doi.org/10.17605/OSF.IO/N3GCH.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/51298.

Bayesian Inference of Hidden Cognitive Performance and Arousal States in Presence of Music

Goal: Poor arousal management may lead to reduced cognitive performance. Specifying a model and decoder to infer the cognitive arousal and performance contributes to arousal regulation via non-invasive actuators such as music. Methods: We employ a Bayesian filtering approach within an expectation-maximization framework to track the hidden states during the [Formula: see text]-back task in the presence of calming and exciting music. We decode the arousal and performance states from the skin conductance and behavioral signals, respectively. We derive an arousal-performance model based on the Yerkes-Dodson law. We design a performance-based arousal decoder by considering the corresponding performance and skin conductance as the observation. Results: The quantified arousal and performance are presented. The existence of Yerkes-Dodson law can be interpreted from the arousal-performance relationship. Findings display higher matrices of performance within the exciting music. Conclusions: The performance-based arousal decoder has a better agreement with the Yerkes-Dodson law. Our study can be implemented in designing non-invasive closed-loop systems.

Sparse Multichannel Decomposition of Electrodermal Activity With Physiological Priors

Goal: Inferring autonomous nervous system (ANS) activity is a challenging issue and has critical applications in stress regulation. Sweat secretions caused by ANS activity influence the electrical conductance of the skin. Therefore, the variations in skin conductance (SC) measurements reflect the sudomotor nerve activity (SMNA) and can be used to infer the underlying ANS activity. These variations are strongly correlated with emotional arousal as well as thermoregulation. However, accurately recovering ANS activity and the corresponding state-space system from a single channel signal is difficult due to artifacts introduced by measurement noise. To minimize the impact of noise on inferring ANS activity, we utilize multiple channels of SC data. Methods: We model skin conductance using a second-order differential equation incorporating a time-shifted sparse impulse train input in combination with independent cubic basis spline functions. Finally, we develop a block coordinate descent method for SC signal decomposition by employing a generalized cross-validation sparse recovery approach while including physiological priors. Results: We analyze the experimental data to validate the performance of the proposed algorithm. We demonstrate its capacity to recover the ANS activations, the underlying physiological system parameters, and both tonic and phasic components. Finally, we present an overview of the algorithm's comparative performance under varying conditions and configurations to substantiate its ability to accurately model ANS activity. Our results show that our algorithm performs better in terms of multiple metrics like noise performance, AUC score, the goodness of fit of reconstructed signal, and lower missing impulses compared with the single channel decomposition approach. Conclusion: In this study, we highlight the challenges and benefits of concurrent decomposition and deconvolution of multichannel SC signals.

Regulation of brain cognitive states through auditory, gustatory, and olfactory stimulation with wearable monitoring

Inspired by advances in wearable technologies, we design and perform human-subject experiments. We aim to investigate the effects of applying safe actuation (i.e., auditory, gustatory, and olfactory) for the purpose of regulating cognitive arousal and enhancing the performance states. In two proposed experiments, subjects are asked to perform a working memory experiment called n-back tasks. Next, we incorporate listening to different types of music, drinking coffee, and smelling perfume as safe actuators. We employ signal processing methods to seamlessly infer participants' brain cognitive states. The results demonstrate the effectiveness of the proposed safe actuation in regulating the arousal state and enhancing performance levels. Employing only wearable devices for human monitoring and using safe actuation intervention are the key components of the proposed experiments. Our dataset fills the existing gap of the lack of publicly available datasets for the self-management of internal brain states using wearable devices and safe everyday actuators. This dataset enables further machine learning and system identification investigations to facilitate future smart work environments. This would lead us to the ultimate idea of developing practical automated personalized closed-loop architectures for managing internal brain states and enhancing the quality of life.

A Point-Process Approach for Tracking Valence using a Respiration Belt

Emotional valence is difficult to be inferred since it is related to several psychological factors and is affected by inter- and intra-subject variability. Changes in emotional valence have been found to cause a physiological response in respiration signals. In this study, we propose a state-space model and decode the valence by analyzing a person's respiration pattern. Particularly, we generate a binary point process based on features that are indicative of changes in respiration pattern as a result of an emotional valence response. High valence is typically associated with faster and deeper breathing. As a result, (i)depth of breath, (ii)rate of respiration, and (iii) breathing cycle time are indicators of high valence and used to generate the binary point process representing underlying neural stimuli associated with changes in valence. We utilize an expectation-maximization (EM) framework to decode a hidden valence state and the associated valence index. This predicted valence state is compared to self-reported valence ratings to optimize the parameters and determine the accuracy of the model. The accuracy of the model in predicting high and low valence events is found to be 77% and 73%, respectively. Our study can be applied towards the long term analysis of valence. Additionally, it has applications in a closed-loop system procedures and wearable design paradigm to track and regulate the emotional valence.

Unraveling the universality of chemical fear communication: evidence from behavioral, genetic, and chemical analyses

Abundant evidence indicates that humans can communicate threat-related information to conspecifics through their body odors. However, prior research has been primarily conducted on Western (WEIRD) samples. In this study, we aimed to investigate whether threat-related information can be transmitted by individuals of East Asian descent who carry a single-nucleotide polymorphism (SNP) 538G → A in the ABCC11 gene, which significantly reduces (noticeable) body odor. To examine this, we recruited 18 self-identified male East Asian AA-homozygotes and 18 self-identified male Western individuals who were carriers of the functional G-allele. We collected samples of their fear-related and neutral body odors. Subsequently, we conducted a double-blind behavioral experiment in which we presented these samples to 69 self-identified female participants of Western Caucasian and East Asian backgrounds. The participants were asked to rate faces that were morphed between expressions of fear and disgust. Notably, despite the "odorless" phenotypical expression of the ABCC11-mutation in East Asians, their fear odor caused a perceptual fear bias in both East Asian and Caucasian receivers. This finding leaves open the possibility of universal fear chemosignaling. Additionally, we conducted exploratory chemical analysis to gain initial insights into the chemical composition of the body odors presented. In a subsequent pre-registered behavioral study (N = 33), we found that exposure to hexadecanoic acid, an abundant compound in the fear and neutral body odor samples, was sufficient to reproduce the observed behavioral effects. While exploratory, these findings provide insight into how specific chemical components can drive chemical fear communication.

Machine Learning Techniques for Arousal Classification from Electrodermal Activity: A Systematic Review

This article introduces a systematic review on arousal classification based on electrodermal activity (EDA) and machine learning (ML). From a first set of 284 articles searched for in six scientific databases, fifty-nine were finally selected according to various criteria established. The systematic review has made it possible to analyse all the steps to which the EDA signals are subjected: acquisition, pre-processing, processing and feature extraction. Finally, all ML techniques applied to the features of these signals for arousal classification have been studied. It has been found that support vector machines and artificial neural networks stand out within the supervised learning methods given their high-performance values. In contrast, it has been shown that unsupervised learning is not present in the detection of arousal through EDA. This systematic review concludes that the use of EDA for the detection of arousal is widely spread, with particularly good results in classification with the ML methods found.

An unsupervised automated paradigm for artifact removal from electrodermal activity in an uncontrolled clinical setting

Objective. Electrodermal activity (EDA) reflects sympathetic nervous system activity through sweating-related changes in skin conductance and could be used in clinical settings in which patients cannot self-report pain, such as during surgery or when in a coma. To enable EDA data to be used robustly in clinical settings, we need to develop artifact detection and removal frameworks that can handle the types of interference experienced in clinical settings while salvaging as much useful information as possible.Approach. In this study, we collected EDA data from 70 subjects while they were undergoing surgery in the operating room. We then built a fully automated artifact removal framework to remove the heavy artifacts that resulted from the use of surgical electrocautery during the surgery and compared it to two existing state-of-the-art methods for artifact removal from EDA data. This automated framework consisted of first utilizing three unsupervised machine learning methods for anomaly detection, and then customizing the threshold to separate artifact for each data instance by taking advantage of the statistical properties of the artifact in that data instance. We also created simulated surgical data by introducing artifacts into cleaned surgical data and measured the performance of all three methods in removing it.Main results. Our method achieved the highest overall accuracy and precision and lowest overall error on simulated data. One of the other methods prioritized high sensitivity while sacrificing specificity and precision, while the other had low sensitivity, high error, and left behind several artifacts. These results were qualitatively similar between the simulated data instances and operating room data instances.Significance. Our framework allows for robust removal of heavy artifact from EDA data in clinical settings such as surgery, which is the first step to enable clinical integration of EDA as part of standard monitoring.