Brain Responses to Emotional Faces in Natural Settings: A Wireless Mobile EEG Recording Study

Machine learning and EEG can classify passive viewing of discrete categories of visual stimuli but not the observation of pain

Previous studies have demonstrated the potential of machine learning (ML) in classifying physical pain from non-pain states using electroencephalographic (EEG) data. However, the application of ML to EEG data to categorise the observation of pain versus non-pain images of human facial expressions or scenes depicting pain being inflicted has not been explored. The present study aimed to address this by training Random Forest (RF) models on cortical event-related potentials (ERPs) recorded while participants passively viewed faces displaying either pain or neutral expressions, as well as action scenes depicting pain or matched non-pain (neutral) scenarios. Ninety-one participants were recruited across three samples, which included a model development group (n = 40) and a cross-subject validation group (n = 51). Additionally, 25 participants from the model development group completed a second experimental session, providing a within-subject temporal validation sample. The analysis of ERPs revealed an enhanced N170 component in response to faces compared to action scenes. Moreover, an increased late positive potential (LPP) was observed during the viewing of pain scenes compared to neutral scenes. Additionally, an enhanced P3 response was found when participants viewed faces displaying pain expressions compared to neutral expressions. Subsequently, three RF models were developed to classify images into faces and scenes, neutral and pain scenes, and neutral and pain expressions. The RF model achieved classification accuracies of 75%, 64%, and 69% for cross-validation, cross-subject, and within-subject classifications, respectively, along with reasonably calibrated predictions for the classification of face versus scene images. However, the RF model was unable to classify pain versus neutral stimuli above chance levels when presented with subsequent tasks involving images from either category. These results expand upon previous findings by externally validating the use of ML in classifying ERPs related to different categories of visual images, namely faces and scenes. The results also indicate the limitations of ML in distinguishing pain and non-pain connotations using ERP responses to the passive viewing of visually similar images.

Moving beyond the lab: investigating empathy through the Empirical 5E approach

Empathy is a complex and multifaceted phenomenon that plays a crucial role in human social interactions. Recent developments in social neuroscience have provided valuable insights into the neural underpinnings and bodily mechanisms underlying empathy. This methodology often prioritizes precision, replicability, internal validity, and confound control. However, fully understanding the complexity of empathy seems unattainable by solely relying on artificial and controlled laboratory settings, while overlooking a comprehensive view of empathy through an ecological experimental approach. In this article, we propose articulating an integrative theoretical and methodological framework based on the 5E approach (the "E"s stand for embodied, embedded, enacted, emotional, and extended perspectives of empathy), highlighting the relevance of studying empathy as an active interaction between embodied agents, embedded in a shared real-world environment. In addition, we illustrate how a novel multimodal approach including mobile brain and body imaging (MoBi) combined with phenomenological methods, and the implementation of interactive paradigms in a natural context, are adequate procedures to study empathy from the 5E approach. In doing so, we present the Empirical 5E approach (E5E) as an integrative scientific framework to bridge brain/body and phenomenological attributes in an interbody interactive setting. Progressing toward an E5E approach can be crucial to understanding empathy in accordance with the complexity of how it is experienced in the real world.

Neuroergonomics on the Go: An Evaluation of the Potential of Mobile EEG for Workplace Assessment and Design

OBJECTIVE

We demonstrate and discuss the use of mobile electroencephalogram (EEG) for neuroergonomics. Both technical state of the art as well as measures and cognitive concepts are systematically addressed.

BACKGROUND

Modern work is increasingly characterized by information processing. Therefore, the examination of mental states, mental load, or cognitive processing during work is becoming increasingly important for ergonomics.

RESULTS

Mobile EEG allows to measure mental states and processes under real live conditions. It can be used for various research questions in cognitive neuroergonomics. Besides measures in the frequency domain that have a long tradition in the investigation of mental fatigue, task load, and task engagement, new approaches-like blink-evoked potentials-render event-related analyses of the EEG possible also during unrestricted behavior.

CONCLUSION

Mobile EEG has become a valuable tool for evaluating mental states and mental processes on a highly objective level during work. The main advantage of this technique is that working environments don't have to be changed while systematically measuring brain functions at work. Moreover, the workflow is unaffected by such neuroergonomic approaches.

WildLab: A naturalistic free viewing experiment reveals previously unknown electroencephalography signatures of face processing

Neural mechanisms of face perception are predominantly studied in well-controlled experimental settings that involve random stimulus sequences and fixed eye positions. Although powerful, the employed paradigms are far from what constitutes natural vision. Here, we demonstrate the feasibility of ecologically more valid experimental paradigms using natural viewing behaviour, by combining a free viewing paradigm on natural scenes, free of photographer bias, with advanced data processing techniques that correct for overlap effects and co-varying non-linear dependencies of multiple eye movement parameters. We validate this approach by replicating classic N170 effects in neural responses, triggered by fixation onsets (fixation event-related potentials [fERPs]). Importantly, besides finding a strong correlation between both experiments, our more natural stimulus paradigm yielded smaller variability between subjects than the classic setup. Moving beyond classic temporal and spatial effect locations, our experiment furthermore revealed previously unknown signatures of face processing: This includes category-specific modulation of the event-related potential (ERP)'s amplitude even before fixation onset, as well as adaptation effects across subsequent fixations depending on their history.

Towards real-world neuroscience using mobile EEG and augmented reality

Our visual environment impacts multiple aspects of cognition including perception, attention and memory, yet most studies traditionally remove or control the external environment. As a result, we have a limited understanding of neurocognitive processes beyond the controlled lab environment. Here, we aim to study neural processes in real-world environments, while also maintaining a degree of control over perception. To achieve this, we combined mobile EEG (mEEG) and augmented reality (AR), which allows us to place virtual objects into the real world. We validated this AR and mEEG approach using a well-characterised cognitive response-the face inversion effect. Participants viewed upright and inverted faces in three EEG tasks (1) a lab-based computer task, (2) walking through an indoor environment while seeing face photographs, and (3) walking through an indoor environment while seeing virtual faces. We find greater low frequency EEG activity for inverted compared to upright faces in all experimental tasks, demonstrating that cognitively relevant signals can be extracted from mEEG and AR paradigms. This was established in both an epoch-based analysis aligned to face events, and a GLM-based approach that incorporates continuous EEG signals and face perception states. Together, this research helps pave the way to exploring neurocognitive processes in real-world environments while maintaining experimental control using AR.

An ERP study on facial emotion processing in young people with subjective memory complaints

Subjective memory complaints (SMCs) are commonly related to aging, but they are also presented by young adults. Their neurophysiological mechanisms are not thoroughly understood, although some aspects related to affective state have been mentioned. Here, we investigated whether facial emotion processing is different in young people with (n = 41) and without (n = 39) SMCs who were exposed to positive, negative, and neutral faces, by recording the event-related potential (ERP) activity. From the ERP activity, the N170 (an index of face processing) and the LPP (an index of motivated attention) components were extracted. Regarding the N170, results showed less amplitude for positive and neutral faces in the participants with SMCs than in those without SMCs. Moreover, women with SMCs displayed longer latencies for neutral faces than women without SMCs. No significant differences were found between the groups in the LPP component. Together, our findings suggest deficits in an early stage of facial emotion processing in young people with SMCs, and they emphasize the importance of further examining affective dimensions.

Comparison of Scalp ERP to Faces in Macaques and Humans

Face recognition is an essential activity of social living, common to many primate species. Underlying processes in the brain have been investigated using various techniques and compared between species. Functional imaging studies have shown face-selective cortical regions and their degree of correspondence across species. However, the temporal dynamics of face processing, particularly processing speed, are likely different between them. Across sensory modalities activation of primary sensory cortices in macaque monkeys occurs at about 3/5 the latency of corresponding activation in humans, though this human simian difference may diminish or disappear in higher cortical regions. We recorded scalp event-related potentials (ERPs) to presentation of faces in macaques and estimated the peak latency of ERP components. Comparisons of latencies between macaques (112 ms) and humans (192 ms) suggested that the 3:5 ratio could be preserved in higher cognitive regions of face processing between those species.

Investigating real-life emotions in romantic couples: a mobile EEG study

The neural basis of emotional processing has been largely investigated in constrained spatial environments such as stationary EEGs or fMRI scanners using highly artificial stimuli like standardized pictures depicting emotional scenes. Typically, such standardized experiments have low ecological validity and it remains unclear whether their results reflect neuronal processing in real-life affective situations at all. Critically, emotional situations do not only encompass the perception of emotions, but also behavioral components associated with them. In this study, we aimed to investigate real-life emotions by recording couples in their homes using mobile EEG technology during embracing, kissing and emotional speech. We focused on asymmetries in affective processing as emotions have been demonstrated to be strongly lateralized in the brain. We found higher alpha and beta power asymmetry during kissing and embracing on frontal electrodes during emotional kisses and speech compared to a neutral control condition indicative of stronger left-hemispheric activation. In contrast, we found lower alpha power asymmetry at parieto-occipital electrode sites in the emotional compared to the neutral condition indicative of stronger right-hemispheric activation. Our findings for alpha power asymmetries are in line with models of emotional lateralization that postulate a valence-specific processing over frontal cortices and right-hemispheric dominance in emotional processing in parieto-occipital regions. In contrast, beta power asymmetries pointed more towards valence-specific processing indicating that, while alpha and beta frequencies seem to be functionally associated, they are not reflecting identical cognitive processing.

Implicit Theory of Mind under realistic social circumstances measured with mobile eye-tracking

Recently, there has been a debate whether implicit Theory of Mind can be reliably measured using anticipatory looking tasks. Previous anticipatory looking paradigms used video stimuli to measure implicit Theory of Mind; however, numerous replications of these paradigms were unsuccessful. This lack of replications may be due to video stimuli not being sufficiently engaging. As Theory of Mind is an inherently social phenomenon, robust evidence might only be observed in a real social situation. Therefore, the current preregistered study aimed to test anticipatory looking with real-life social stimuli. A mobile eye-tracker was used to measure gaze patterns indicative of Theory of Mind while participants observed a real-life interaction of an experimenter and a confederate. The realistic scenario did not provide clear evidence for implicit Theory of Mind. Furthermore, anticipatory looking behavior did not reliably occur during familiarization trials, in line with previous research. However, looking patterns were slightly more in line with belief tracking than in some more controlled studies using video stimuli. In general, implicit Theory of Mind was not reliably reflected in anticipatory looking patterns even if they were measured in realistic social situations. This questions the suitability of anticipatory looking measures for implicit Theory of Mind.

Cortical Patterns of Pleasurable Musical Chills Revealed by High-Density EEG

Music has the capacity to elicit strong positive feelings in humans by activating the brain's reward system. Because group emotional dynamics is a central concern of social neurosciences, the study of emotion in natural/ecological conditions is gaining interest. This study aimed to show that high-density EEG (HD-EEG) is able to reveal patterns of cerebral activities previously identified by fMRI or PET scans when the subject experiences pleasurable musical chills. We used HD-EEG to record participants (11 female, 7 male) while listening to their favorite pleasurable chill-inducing musical excerpts; they reported their subjective emotional state from low pleasure up to chills. HD-EEG results showed an increase of theta activity in the prefrontal cortex when arousal and emotional ratings increased, which are associated with orbitofrontal cortex activation localized using source localization algorithms. In addition, we identified two specific patterns of chills: a decreased theta activity in the right central region, which could reflect supplementary motor area activation during chills and may be related to rhythmic anticipation processing, and a decreased theta activity in the right temporal region, which may be related to musical appreciation and could reflect the right superior temporal gyrus activity. The alpha frontal/prefrontal asymmetry did not reflect the felt emotional pleasure, but the increased frontal beta to alpha ratio (measure of arousal) corresponded to increased emotional ratings. These results suggest that EEG may be a reliable method and a promising tool for the investigation of group musical pleasure through musical reward processing.

Eye contact in active and passive viewing: Event-related brain potential evidence from a combined eye tracking and EEG study

Eye contact is a salient social cue, which is assumed to influence already early neural correlates of face perception. Specifically, the N170 component of the event-related potential (ERP) has often been found to be larger for faces with an averted gaze as compared to faces that directly look at the observer. In most existing ERP studies, effects of eye contact were investigated under comparatively artificial conditions where participants were instructed to maintain a steady fixation while they passively observed gaze changes in the stimulus face. It is therefore unclear to what extent neural correlates of eye contact generalize to more naturalistic situations that involve a continuous interplay between directed and averted gaze between the communication partners. To start bridging this gap, the present study compared the passive viewing of gaze changes to an active condition in which the participant's own gaze (measured online with an eye tracker) interacted with the gaze position of a continuously presented stimulus face. We also investigated whether eye contact effects were modulated by the face's emotional expression. In both the passive and the active viewing condition, N170 amplitudes were larger when the gaze of the stimulus faces was averted rather than directed towards the participant. Furthermore, eye contact decreased P300 amplitudes in both conditions. The emotional expression of the face also modulated the N170, but this effect did not interact with that of gaze direction. We conclude that the neural correlates of gaze perception during active gaze interactions are comparable to those found during passive viewing, encouraging the further study of eye contact effects in more naturalistic settings.

Face Selective Neural Activity: Comparisons Between Fixed and Free Viewing

Event Related Potentials (ERPs) are widely used to study category-selective EEG responses to visual stimuli, such as the face-selective N170 component. Typically, this is done by flashing stimuli at the point of static gaze fixation. While allowing for good experimental control, these paradigms ignore the dynamic role of eye-movements in natural vision. Fixation-related potentials (FRPs), obtained using simultaneous EEG and eye-tracking, overcome this limitation. Various studies have used FRPs to study processes such as lexical processing, target detection and attention allocation. The goal of this study was to carefully compare face-sensitive activity time-locked to an abrupt stimulus onset at fixation, with that time-locked to a self-generated fixation on a stimulus. Twelve participants participated in three experimental conditions: Free-viewing (FRPs), Cued-viewing (FRPs) and Control (ERPs). We used a multiple regression approach to disentangle overlapping activity components. Our results show that the N170 face-effect is evident for the first fixation on a stimulus, whether it follows a self-generated saccade or stimulus appearance at fixation point. The N170 face-effect has similar topography across viewing conditions, but there were major differences within each stimulus category. We ascribe these differences to an overlap of the fixation-related lambda response and the N170. We tested the plausibility of this account using dipole simulations. Finally, the N170 exhibits category-specific adaptation in free viewing. This study establishes the comparability of the free-viewing N170 face-effect with the classic event-related effect, while highlighting the importance of accounting for eye-movement related effects.

Post-Saccadic Face Processing Is Modulated by Pre-Saccadic Preview: Evidence from Fixation-Related Potentials

Humans actively sample their environment with saccadic eye movements to bring relevant information into high-acuity foveal vision. Despite being lower in resolution, peripheral information is also available before each saccade. How the pre-saccadic extrafoveal preview of a visual object influences its post-saccadic processing is still an unanswered question. The current study investigated this question by simultaneously recording behavior and fixation-related brain potentials while human subjects made saccades to face stimuli. We manipulated the relationship between pre-saccadic "previews" and post-saccadic images to explicitly isolate the influences of the former. Subjects performed a gender discrimination task on a newly foveated face under three preview conditions: scrambled face, incongruent face (different identity from the foveated face), and congruent face (same identity). As expected, reaction times were faster after a congruent-face preview compared with a scrambled-face preview. Importantly, intact face previews (either incongruent or congruent) resulted in a massive reduction of post-saccadic neural responses. Specifically, we analyzed the classic face-selective N170 component at occipitotemporal electroencephalogram electrodes, which was still present in our experiments with active looking. However, the post-saccadic N170 was strongly attenuated following intact-face previews compared with the scrambled condition. This large and long-lasting decrease in evoked activity is consistent with a trans-saccadic mechanism of prediction that influences category-specific neural processing at the start of a new fixation. These findings constrain theories of visual stability and show that the extrafoveal preview methodology can be a useful tool to investigate its underlying mechanisms.SIGNIFICANCE STATEMENT Neural correlates of object recognition have traditionally been studied by flashing stimuli to the central visual field. This procedure differs in fundamental ways from natural vision, where viewers actively sample the environment with eye movements and also obtain a low-resolution preview of soon-to-be-fixated objects. Here we show that the N170, a classic electrophysiological marker of the structural encoding of faces, also occurs during a more natural viewing condition but is strongly reduced due to extrafoveal preprocessing (preview benefit). Our results therefore highlight the importance of peripheral vision during trans-saccadic processing in building a coherent and stable representation of the world around us.

Neural underpinnings of value-guided choice during auction tasks: An eye-fixation related potentials study

Values are attributed to goods during free viewing of objects which entails multi- and trans-saccadic cognitive processes. Using electroencephalographic eye-fixation related potentials, the present study investigated how neural signals related to value-guided choice evolved over time when viewing household and office products during an auction task. Participants completed a Becker-DeGroot-Marschak auction task whereby half of the stimuli were presented in either a free or forced bid protocol to obtain willingness-to-pay. Stimuli were assigned to three value categories of low, medium and high value based on subjective willingness-to-pay. Eye fixations were organised into five 800 ms time-bins spanning the objects total viewing time. Independent component analysis was applied to eye-fixation related potentials. One independent component (IC) was found to represent fixations for high value products with increased activation over the left parietal region of the scalp. An IC with a spatial maximum over a fronto-central region of the scalp coded the intermediate values. Finally, one IC displaying activity that extends over the right frontal scalp region responded to intermediate- and low-value items. Each of these components responded early on during viewing an object and remained active over the entire viewing period, both during free and forced bid trials. Results suggest that the subjective value of goods are encoded using sets of brain activation patterns which are tuned to respond uniquely to either low, medium, or high values. Data indicates that the right frontal region of the brain responds to low and the left frontal region to high values. Values of goods are determined at an early point in the decision making process and carried for the duration of the decision period via trans-saccadic processes.

Optimizing the ICA-based removal of ocular EEG artifacts from free viewing experiments

Combining EEG with eye-tracking is a promising approach to study neural correlates of natural vision, but the resulting recordings are also heavily contaminated by activity of the eye balls, eye lids, and extraocular muscles. While Independent Component Analysis (ICA) is commonly used to suppress these ocular artifacts, its performance under free viewing conditions has not been systematically evaluated and many published reports contain residual artifacts. Here I evaluated and optimized ICA-based correction for two tasks with unconstrained eye movements: visual search in images and sentence reading. In a first step, four parameters of the ICA pipeline were varied orthogonally: the (1) high-pass and (2) low-pass filter applied to the training data, (3) the proportion of training data containing myogenic saccadic spike potentials (SP), and (4) the threshold for eye tracker-based component rejection. In a second step, the eye-tracker was used to objectively quantify the correction quality of each ICA solution, both in terms of undercorrection (residual artifacts) and overcorrection (removal of neurogenic activity). As a benchmark, results were compared to those obtained with an alternative spatial filter, Multiple Source Eye Correction (MSEC). With commonly used settings, Infomax ICA not only left artifacts in the data, but also distorted neurogenic activity during eye movement-free intervals. However, correction results could be strongly improved by training the ICA on optimally filtered data in which SPs were massively overweighted. With optimized procedures, ICA removed virtually all artifacts, including the SP and its associated spectral broadband artifact from both viewing paradigms, with little distortion of neural activity. It also outperformed MSEC in terms of SP correction. Matlab code is provided.

Tracking Economic Value of Products in Natural Settings: A Wireless EEG Study

Economic decision making refers to the process of individuals translating their preference into subjective value (SV). Little is known about the dynamics of the neural processes that underpin this form of value-based decision making and no studies have investigated these processes outside of controlled laboratory settings. The current study investigated the spatio-temporal dynamics that accompany economic valuation of products using mobile electroencephalography (EEG) and eye tracking techniques. Participants viewed and rated images of household products in a gallery setting while EEG and eye tracking data were collected wirelessly. A Becker-DeGroot-Marschak (BDM) auction task was subsequently used to quantify the individual's willingness to pay (WTP) for each product. WTP was used to classify products into low, low medium, high medium and high economic value conditions. Eye movement related potentials (EMRP) were examined, and independent component analysis (ICA) was used to separate sources of activity from grand averaged EEG data. Four independent components (ICs) of EMRPs were modulated by WTP (i.e., SV) in the latency range of 150-250 ms. Of the four value-sensitive ICs, one IC displayed enhanced amplitude for all value conditions excluding low value, and another IC presented enhanced amplitude for low value products only. The remaining two value-sensitive ICs resolved inter-mediate levels of SV. Our study quantified, for the first time, the neural processes involved in economic value based decisions in a natural setting. Results suggest that multiple spatio-temporal brain activation patterns mediate the attention and aversion of products which could reflect an early valuation system. The EMRP parietal P200 component could reflect an attention allocation mechanism that separates the lowest-value products (IC7) from products of all other value (IC4), suggesting that low-value items are categorized early on as being aversive. While none of the ICs showed linear amplitude changes that parallel SV's of products, results suggest that a combination of multiple components may sub-serve a fine-grained resolution of the SV of products.