Inference of human affective states from psychophysiological measurements extracted under ecologically valid conditions

Utilising Emotion Monitoring for Developing Music Interventions for People with Dementia: A State-of-the-Art Review

The demand for smart solutions to support people with dementia (PwD) is increasing. These solutions are expected to assist PwD with their emotional, physical, and social well-being. At the moment, state-of-the-art works allow for the monitoring of physical well-being; however, not much attention is delineated for monitoring the emotional and social well-being of PwD. Research on emotion monitoring can be combined with research on the effects of music on PwD given its promising effects. More specifically, knowledge of the emotional state allows for music intervention to alleviate negative emotions by eliciting positive emotions in PwD. In this direction, the paper conducts a state-of-the-art review on two aspects: (i) the effect of music on PwD and (ii) both wearable and non-wearable sensing systems for emotional state monitoring. After outlining the application of musical interventions for PwD, including emotion monitoring sensors and algorithms, multiple challenges are identified. The main findings include a need for rigorous research approaches for the development of adaptable solutions that can tackle dynamic changes caused by the diminishing cognitive abilities of PwD with a focus on privacy and adoption aspects. By addressing these requirements, advancements can be made in harnessing music and emotion monitoring for PwD, thereby facilitating the creation of more resilient and scalable solutions to aid caregivers and PwD.

VR for Studying the Neuroscience of Emotional Responses

Emotions are frequently considered as the driving force of behavior, and psychopathology is often characterized by aberrant emotional responding. Emotional states are reflected on a cognitive-verbal, physiological-humoral, and motor-behavioral level but to date, human research lacks an experimental protocol for a comprehensive and ecologically valid characterization of such emotional states. Virtual reality (VR) might help to overcome this situation by allowing researchers to study mental processes and behavior in highly controlled but reality-like laboratory settings. In this chapter, we first elucidate the role of presence and immersion as requirements for eliciting emotional states in a virtual environment and discuss different VR methods for emotion induction. We then consider the organization of emotional states on a valence continuum (i.e., from negative to positive) and on this basis discuss the use of VR to study threat processing and avoidance as well as reward processing and approach behavior. Although the potential of VR has not been fully realized in laboratory and clinical settings yet, this technological tool can open up new avenues to better understand the neurobiological mechanisms of emotional responding in healthy and pathological conditions.

Wearable Systems for Home Monitoring Healthcare: The Photoplethysmography Success Pros and Cons

The widespread use of remote technology has moved medical care services into individuals' homes. In this perspective, the ubiquitous computing research proposes self-management and remote monitoring to help patients with healthcare in low-cost everyday home usage systems based on the latest technological advances in sensors, communication, and portability. This work analyzes recent publications on the paradigm of continuous monitoring through wearable and portable systems, focusing on photoplethysmography (PPG) advances and referencing the current systematic study proposed by Fine et al. The study revised the literature highlighting the pros and cons of using the PPG system for fitness, wellbeing, and medical devices. However, future works should focus on the standardization of the practical use and assessment of the quality of the PPGs' output. For clinical parameter extraction methodology in terms of biological sites of application and signal processing methods, PPG is the most convenient and widely used system potentially suitable for the decentralized paradigm of continuous monitoring healthcare concepts.

Emotion Recognition With Knowledge Graph Based on Electrodermal Activity

Electrodermal activity (EDA) sensor is emerging non-invasive equipment in affect detection research, which is used to measure electrical activities of the skin. Knowledge graphs are an effective way to learn representation from data. However, few studies analyzed the effect of knowledge-related graph features with physiological signals when subjects are in non-similar mental states. In this paper, we propose a model using deep learning techniques to classify the emotional responses of individuals acquired from physiological datasets. We aim to improve the execution of emotion recognition based on EDA signals. The proposed framework is based on observed gender and age information as embedding feature vectors. We also extract time and frequency EDA features in line with cognitive studies. We then introduce a sophisticated weighted feature fusion method that combines knowledge embedding feature vectors and statistical feature (SF) vectors for emotional state classification. We finally utilize deep neural networks to optimize our approach. Results obtained indicated that the correct combination of Gender-Age Relation Graph (GARG) and SF vectors improve the performance of the valence-arousal emotion recognition system by 4 and 5% on PAFEW and 3 and 2% on DEAP datasets.

Contextual modulation of conditioned responses in humans: A review on virtual reality studies

Conditioned response (CRs) triggered by stimuli predicting aversive consequences have been confirmed across various species including humans, and were found to be exaggerated in anxious individuals and anxiety disorder patients. Importantly, contextual information may strongly modulate such conditioned responses (CR), however, there are several methodological boundaries in the translation of animal findings to humans, and from healthy individuals to patients. Virtual Reality (VR) is a useful technological tool for overcoming such boundaries. In this review, we summarize and evaluate human VR conditioning studies exploring the role of the context as conditioned stimulus or occasion setter for CRs. We observe that VR allows successful acquisition of conditioned anxiety and conditioned fear in response to virtual contexts and virtual cues, respectively. VR studies also revealed that spatial or temporal contextual information determine whether conditioned anxiety and conditioned fear become extinguished and/or return. Novel contexts resembling the threatening context foster conditioned fear but not conditioned anxiety, suggesting distinct context-related generalization processes. We conclude VR contexts are able to strongly modulate CRs and therefore allow a comprehensive investigation of the modulatory role of the context over CR in humans leading to conclusions relevant for non-VR and clinical studies.

Extended Reality for the Clinical, Affective, and Social Neurosciences

Brain science research often involves the use of low-dimensional tools and stimuli that lack several of the potentially valuable features of everyday activities and interactions. Although this research has provided important information about cognitive, affective, and social processes for both clinical and nonclinical populations, there is growing interest in high-dimensional simulations that extend reality. These high-dimensional simulations involve dynamic stimuli presented serially or concurrently to permit the assessment and training of perceivers' integrative processes over time. Moreover, high-dimensional simulation platforms can contextually restrain interpretations of cues about a target's internal states. Extended reality environments extend assessment and training platforms that balance experimental control with emotionally engaging background narratives aimed at extending the affective experience and social interactions. Herein, we highlight the promise of extended reality platforms for greater ecological validity in the clinical, affective, and social neurosciences.

Using Psychophysiological Sensors to Assess Mental Workload During Web Browsing

Knowledge of the mental workload induced by a Web page is essential for improving users’ browsing experience. However, continuously assessing the mental workload during a browsing task is challenging. To address this issue, this paper leverages the correlation between stimuli and physiological responses, which are measured with high-frequency, non-invasive psychophysiological sensors during very short span windows. An experiment was conducted to identify levels of mental workload through the analysis of pupil dilation measured by an eye-tracking sensor. In addition, a method was developed to classify mental workload by appropriately combining different signals (electrodermal activity (EDA), electrocardiogram, photoplethysmo-graphy (PPG), electroencephalogram (EEG), temperature and pupil dilation) obtained with non-invasive psychophysiological sensors. The results show that the Web browsing task involves four levels of mental workload. Also, by combining all the sensors, the efficiency of the classification reaches 93.7%.

Feasibility of an electrodermal activity ring prototype as a research tool

Electrodermal activity is an indicator of sympathetic activation and a useful tool for investigating psychological and physiological arousal. Novel wearable skin conductivity sensors offer portable low-cost solutions for long-term monitoring. In this article we compare the similarity of signals between a prototype of the wearable Moodmetric EDA Ring and a laboratory-grade skin conductance sensor in a psychophysiological experiment. The similarity of the signals was estimated by calculating the cosine distance between phasic features extracted from decomposed signals. The similarity was on average 83.3% ± 16.4%. The compound error of the decomposition process was also investigated and no systematic bias was observed towards either device. We conclude that the prototype ring is a promising device for ecologically valid field studies.

The Affective Slider: A Digital Self-Assessment Scale for the Measurement of Human Emotions

Self-assessment methods are broadly employed in emotion research for the collection of subjective affective ratings. The Self-Assessment Manikin (SAM), a pictorial scale developed in the eighties for the measurement of pleasure, arousal, and dominance, is still among the most popular self-reporting tools, despite having been conceived upon design principles which are today obsolete. By leveraging on state-of-the-art user interfaces and metacommunicative pictorial representations, we developed the Affective Slider (AS), a digital self-reporting tool composed of two slider controls for the quick assessment of pleasure and arousal. To empirically validate the AS, we conducted a systematic comparison between AS and SAM in a task involving the emotional assessment of a series of images taken from the International Affective Picture System (IAPS), a database composed of pictures representing a wide range of semantic categories often used as a benchmark in psychological studies. Our results show that the AS is equivalent to SAM in the self-assessment of pleasure and arousal, with two added advantages: the AS does not require written instructions and it can be easily reproduced in latest-generation digital devices, including smartphones and tablets. Moreover, we compared new and normative IAPS ratings and found a general drop in reported arousal of pictorial stimuli. Not only do our results demonstrate that legacy scales for the self-report of affect can be replaced with new measurement tools developed in accordance to modern design principles, but also that standardized sets of stimuli which are widely adopted in research on human emotion are not as effective as they were in the past due to a general desensitization towards highly arousing content.

Virtual Reality for Enhanced Ecological Validity and Experimental Control in the Clinical, Affective and Social Neurosciences

An essential tension can be found between researchers interested in ecological validity and those concerned with maintaining experimental control. Research in the human neurosciences often involves the use of simple and static stimuli lacking many of the potentially important aspects of real world activities and interactions. While this research is valuable, there is a growing interest in the human neurosciences to use cues about target states in the real world via multimodal scenarios that involve visual, semantic, and prosodic information. These scenarios should include dynamic stimuli presented concurrently or serially in a manner that allows researchers to assess the integrative processes carried out by perceivers over time. Furthermore, there is growing interest in contextually embedded stimuli that can constrain participant interpretations of cues about a target’s internal states. Virtual reality environments proffer assessment paradigms that combine the experimental control of laboratory measures with emotionally engaging background narratives to enhance affective experience and social interactions. The present review highlights the potential of virtual reality environments for enhanced ecological validity in the clinical, affective, and social neurosciences.

Electrodermal activity analysis during affective haptic elicitation

This paper investigates how the autonomic nervous system dynamics, quantified through the analysis of the electrodermal activity (EDA), is modulated according to affective haptic stimuli. Specifically, a haptic display able to convey caress-like stimuli is presented to 32 healthy subjects (16 female). Each stimulus is changed according to six combinations of three velocities and two forces levels of two motors stretching a strip of fabric. Subjects were also asked to score each stimulus in terms of arousal (high/low activation) and valence (pleasant/unpleasant), in agreement with the circumplex model of affect. EDA was processed using a deconvolutive method, separating tonic and phasic components. A statistical analysis was performed in order to identify significant differences in EDA features among force and velocity levels, as well as in their valence and arousal scores. Results show that the simulated caress induced by the haptic display significantly affects the EDA. In detail, the phasic component seems to be inversely related to the valence score. This finding is new and promising, since it can be used, e.g., as an additional cue for haptics design.

Using neurophysiological signals that reflect cognitive or affective state: six recommendations to avoid common pitfalls

Estimating cognitive or affective state from neurophysiological signals and designing applications that make use of this information requires expertise in many disciplines such as neurophysiology, machine learning, experimental psychology, and human factors. This makes it difficult to perform research that is strong in all its aspects as well as to judge a study or application on its merits. On the occasion of the special topic "Using neurophysiological signals that reflect cognitive or affective state" we here summarize often occurring pitfalls and recommendations on how to avoid them, both for authors (researchers) and readers. They relate to defining the state of interest, the neurophysiological processes that are expected to be involved in the state of interest, confounding factors, inadvertently "cheating" with classification analyses, insight on what underlies successful state estimation, and finally, the added value of neurophysiological measures in the context of an application. We hope that this paper will support the community in producing high quality studies and well-validated, useful applications.

Network dynamics with BrainX(3): a large-scale simulation of the human brain network with real-time interaction

BrainX³ is a large-scale simulation of human brain activity with real-time interaction, rendered in 3D in a virtual reality environment, which combines computational power with human intuition for the exploration and analysis of complex dynamical networks. We ground this simulation on structural connectivity obtained from diffusion spectrum imaging data and model it on neuronal population dynamics. Users can interact with BrainX³ in real-time by perturbing brain regions with transient stimulations to observe reverberating network activity, simulate lesion dynamics or implement network analysis functions from a library of graph theoretic measures. BrainX³ can thus be used as a novel immersive platform for exploration and analysis of dynamical activity patterns in brain networks, both at rest or in a task-related state, for discovery of signaling pathways associated to brain function and/or dysfunction and as a tool for virtual neurosurgery. Our results demonstrate these functionalities and shed insight on the dynamics of the resting-state attractor. Specifically, we found that a noisy network seems to favor a low firing attractor state. We also found that the dynamics of a noisy network is less resilient to lesions. Our simulations on TMS perturbations show that even though TMS inhibits most of the network, it also sparsely excites a few regions. This is presumably due to anti-correlations in the dynamics and suggests that even a lesioned network can show sparsely distributed increased activity compared to healthy resting-state, over specific brain areas.