Processing of Facial Expressions in Autism: a Systematic Review of EEG/ERP Evidence

Sex modulation of faces prediction error in the autistic brain

Recent research suggests that autistic females may have superior socio-cognitive abilities compared to autistic males, potentially contributing to underdiagnosis in females. However, it remains unclear whether these differences arise from distinct neurophysiological functioning in autistic males and females. This study addresses this question by presenting 41 autistic and 48 non-autistic adults with a spatially filtered faces oddball paradigm. Analysis of event-related potentials from scalp electroencephalography reveal a neurophysiological profile in autistic females that fell between those of autistic males and non-autistic females, highlighting sex differences in autism from the initial stages of face processing. This finding underscores the urgent need to explore neurophysiological sex differences in autism and encourages efforts toward a better comprehension of compensation mechanism and a clearer definition of what is meant by camouflaging.

A neurophysiological signature of dynamic emotion recognition associated with social communication skills and cortical gamma-aminobutyric acid levels in children

Introduction

Emotion recognition is a core feature of social perception. In particular, perception of dynamic facial emotional expressions is a major feature of the third visual pathway. However, the classical N170 visual evoked signal does not provide a pure correlate of such processing. Indeed, independent component analysis has demonstrated that the N170 component is already active at the time of the P100, and is therefore distorted by early components. Here we implemented, a dynamic face emotional paradigm to isolate a more pure face expression selective N170. We searched for a neural correlate of perception of dynamic facial emotional expressions, by starting with a face baseline from which a facial expression evolved. This allowed for a specific facial expression contrast signal which we aimed to relate with social communication abilities and cortical gamma-aminobutyric acid (GABA) levels.

Methods

We recorded event-related potentials (ERPs) and Magnetic Resonance (MRS) measures in 35 typically developing (TD) children, (10-16 years) sex-matched, during emotion recognition of an avatar morphing/unmorphing from neutral to happy/sad expressions. This task allowed for the elimination of the contribution low-level visual components, in particular the P100, by morphing baseline isoluminant neutral faces into specific expressions, isolating dynamic emotion recognition. Therefore, it was possible to isolate a dynamic face sensitive N170 devoid of interactions with earlier components.

Results

We found delayed N170 and P300, with a hysteresis type of dependence on stimulus trajectory (morphing/unmorphing), with hemispheric lateralization. The delayed N170 is generated by an extrastriate source, which can be related to the third visual pathway specialized in biological motion processing. GABA levels in visual cortex were related with N170 amplitude and latency and predictive of worse social communication performance (SCQ scores). N170 latencies reflected delayed processing speed of emotional expressions and related to worse social communication scores.

Discussion

In sum, we found a specific N170 electrophysiological signature of dynamic face processing related to social communication abilities and cortical GABA levels. These findings have potential clinical significance supporting the hypothesis of a spectrum of social communication abilities and the identification of a specific face-expression sensitive N170 which can potentially be used in the development of diagnostic and intervention tools.

Neural sensitivity to facial identity and facial expression discrimination in adults with autism

The fluent processing of faces can be challenging for autistic individuals. Here, we assessed the neural sensitivity to rapid changes in subtle facial cues in 23 autistic men and 23 age and IQ matched non-autistic (NA) controls using frequency-tagging electroencephalography (EEG). In oddball paradigms examining the automatic and implicit discrimination of facial identity and facial expression, base rate images were presented at 6 Hz, periodically interleaved every fifth image with an oddball image (i.e. 1.2 Hz oddball frequency). These distinctive frequency tags for base rate and oddball stimuli allowed direct and objective quantification of the neural discrimination responses. We found no large differences in the neural sensitivity of participants in both groups, not for facial identity discrimination, nor for facial expression discrimination. Both groups also showed a clear face-inversion effect, with reduced brain responses for inverted versus upright faces. Furthermore, sad faces generally elicited significantly lower neural amplitudes than angry, fearful and happy faces. The only minor group difference is the larger involvement of high-level right-hemisphere visual areas in NA men for facial expression processing. These findings are discussed from a developmental perspective, as they strikingly contrast with robust face processing deficits observed in autistic children using identical EEG paradigms.

A new error-monitoring brain-computer interface based on reinforcement learning for people with autism spectrum disorders

Objective.Brain-computer interfaces (BCIs) are emerging as promising cognitive training tools in neurodevelopmental disorders, as they combine the advantages of traditional computerized interventions with real-time tailored feedback. We propose a gamified BCI based on non-volitional neurofeedback for cognitive training, aiming at reaching a neurorehabilitation tool for application in autism spectrum disorders (ASDs).Approach.The BCI consists of an emotional facial expression paradigm controlled by an intelligent agent that makes correct and wrong actions, while the user observes and judges the agent's actions. The agent learns through reinforcement learning (RL) an optimal strategy if the participant generates error-related potentials (ErrPs) upon incorrect agent actions. We hypothesize that this training approach will allow not only the agent to learn but also the BCI user, by participating through implicit error scrutiny in the process of learning through operant conditioning, making it of particular interest for disorders where error monitoring processes are altered/compromised such as in ASD. In this paper, the main goal is to validate the whole methodological BCI approach and assess whether it is feasible enough to move on to clinical experiments. A control group of ten neurotypical participants and one participant with ASD tested the proposed BCI approach.Main results.We achieved an online balanced-accuracy in ErrPs detection of 81.6% and 77.1%, respectively for two different game modes. Additionally, all participants achieved an optimal RL strategy for the agent at least in one of the test sessions.Significance.The ErrP classification results and the possibility of successfully achieving an optimal learning strategy, show the feasibility of the proposed methodology, which allows to move towards clinical experimentation with ASD participants to assess the effectiveness of the approach as hypothesized.

Automatic Emotion Recognition in Children with Autism: A Systematic Literature Review

The automatic emotion recognition domain brings new methods and technologies that might be used to enhance therapy of children with autism. The paper aims at the exploration of methods and tools used to recognize emotions in children. It presents a literature review study that was performed using a systematic approach and PRISMA methodology for reporting quantitative and qualitative results. Diverse observation channels and modalities are used in the analyzed studies, including facial expressions, prosody of speech, and physiological signals. Regarding representation models, the basic emotions are the most frequently recognized, especially happiness, fear, and sadness. Both single-channel and multichannel approaches are applied, with a preference for the first one. For multimodal recognition, early fusion was the most frequently applied. SVM and neural networks were the most popular for building classifiers. Qualitative analysis revealed important clues on participant group construction and the most common combinations of modalities and methods. All channels are reported to be prone to some disturbance, and as a result, information on a specific symptoms of emotions might be temporarily or permanently unavailable. The challenges of proper stimuli, labelling methods, and the creation of open datasets were also identified.

Multimodal Evidence of Atypical Processing of Eye Gaze and Facial Emotion in Children With Autistic Traits

According to the shared signal hypothesis (SSH) the impact of facial expressions on emotion processing partially depends on whether the gaze is directed toward or away from the observer. In autism spectrum disorder (ASD) several aspects of face processing have been found to be atypical, including attention to eye gaze and the identification of emotional expressions. However, there is little research on how gaze direction affects emotional expression processing in typically developing (TD) individuals and in those with ASD. This question is investigated here in two multimodal experiments. Experiment 1 required processing eye gaze direction while faces differed in emotional expression. Forty-seven children (aged 9–12 years) participated. Their Autism Diagnostic Observation Schedule (ADOS) scores ranged from 0 to 6 in the experiment. Event-related potentials (ERPs) were sensitive to gaze direction and emotion, but emotion processing did not depend on gaze direction. However, for angry faces the gaze direction effect on the N170 amplitude, as typically observed in TD individuals, diminished with increasing ADOS score. For neutral expressions this correlation was not significant. Experiment 2 required explicit emotion classifications in a facial emotion composite task while eye gaze was manipulated incidentally. A group of 22 children with ASD was compared to a propensity score-matched group of TD children (mean age = 13 years). The same comparison was carried out for a subgroup of nine children with ASD who were less trained in social cognition, according to clinician’s report. The ASD group performed overall worse in emotion recognition than the TD group, independently of emotion or gaze direction. However, for disgust expressions, eye tracking data revealed that TD children fixated relatively longer on the eyes of the stimulus face with a direct gaze as compared with averted gaze. In children with ASD we observed no such modulation of fixation behavior as a function of gaze direction. Overall, the present findings from ERPs and eye tracking confirm the hypothesis of an impaired sensitivity to gaze direction in children with ASD or elevated autistic traits, at least for specific emotions. Therefore, we conclude that multimodal investigations of the interaction between emotional processing and stimulus gaze direction are promising to understand the characteristics of individuals differing along the autism trait dimension.

Advances in Electrical Source Imaging: A Review of the Current Approaches, Applications and Challenges

Brain source localization has been consistently implemented over the recent years to elucidate complex brain operations, pairing the high temporal resolution of the EEG with the high spatial estimation of the estimated sources. This review paper aims to present the basic principles of Electrical source imaging (ESI) in the context of the recent progress for solving the forward and the inverse problems, and highlight the advantages and limitations of the different approaches. As such, a synthesis of the current state-of-the-art methodological aspects is provided, offering a complete overview of the present advances with regard to the ESI solutions. Moreover, the new dimensions for the analysis of the brain processes are indicated in terms of clinical and cognitive ESI applications, while the prevailing challenges and limitations are thoroughly discussed, providing insights for future approaches that could help to alleviate methodological and technical shortcomings.

Event-related EEG oscillatory responses elicited by dynamic facial expression

BACKGROUND

Recognition of facial expressions (FEs) plays a crucial role in social interactions. Most studies on FE recognition use static (image) stimuli, even though real-life FEs are dynamic. FE processing is complex and multifaceted, and its neural correlates remain unclear. Transitioning from static to dynamic FE stimuli might help disentangle the neural oscillatory mechanisms underlying face processing and recognition of emotion expression. To our knowledge, we here present the first time-frequency exploration of oscillatory brain mechanisms underlying the processing of dynamic FEs.

RESULTS

Videos of joyful, fearful, and neutral dynamic facial expressions were presented to 18 included healthy young adults. We analyzed event-related activity in electroencephalography (EEG) data, focusing on the delta, theta, and alpha-band oscillations. Since the videos involved a transition from neutral to emotional expressions (onset around 500 ms), we identified time windows that might correspond to face perception initially (time window 1; first TW), and emotion expression recognition subsequently (around 1000 ms; second TW). First TW showed increased power and phase-locking values for all frequency bands. In the first TW, power and phase-locking values were higher in the delta and theta bands for emotional FEs as compared to neutral FEs, thus potentially serving as a marker for emotion recognition in dynamic face processing.

CONCLUSIONS

Our time-frequency exploration revealed consistent oscillatory responses to complex, dynamic, ecologically meaningful FE stimuli. We conclude that while dynamic FE processing involves complex network dynamics, dynamic FEs were successfully used to reveal temporally separate oscillation responses related to face processing and subsequently emotion expression recognition.

Linear and nonlinear analysis of intrinsic mode function after facial stimuli presentation in children with autism spectrum disorder

In this work, a method for classifying Autism Spectrum Disorders (ASD) from typically developing (TD) children is presented using the linear and nonlinear Event-Related Potential (ERP) analysis of the Electro-encephalogram (EEG) signals. The signals were acquired during the presentation of three types of face expression stimuli -happy, fearful and neutral faces. EEGs are first decomposed using the Multivariate Empirical Mode Decomposition (MEMD) method to extract its Intrinsic Mode Functions (IMFs), which provide information about the underlying activities of ERP components. The nonlinear sample entropy (SampEn) features, as well as the standard linear measurements utilizing maximum (Max.), minimum (Min), and standard deviation (Std.), are then extracted from each set of IMFs. These features are then evaluated by the statistical analysis tests and used to construct the input vectors for the Discriminant analysis (DA), Support vector machine (SVM), and k-Nearest Neighbors (kNN) classifiers. Experimental results show that the proposed features can differentiate the ASD and TD children using the happy stimulus dataset with high classification performance for all classifiers that reached 100% accuracy. This result suggests a general deficit in recognizing the positive expression in ASD children. Additionally, we found that the SampEn measurements computed from the alpha and theta bands and the linear features extracted from the delta band can be considered biomarkers for disturbances in Emotional Facial Expression (EFE) processing in ASD children.

Facial Expression Processing Across the Autism-Psychosis Spectra: A Review of Neural Findings and Associations With Adverse Childhood Events

Autism spectrum disorder (ASD) and primary psychosis are classified as distinct neurodevelopmental disorders, yet they display overlapping epidemiological, environmental, and genetic components as well as endophenotypic similarities. For instance, both disorders are characterized by impairments in facial expression processing, a crucial skill for effective social communication, and both disorders display an increased prevalence of adverse childhood events (ACE). This narrative review provides a brief summary of findings from neuroimaging studies investigating facial expression processing in ASD and primary psychosis with a focus on the commonalities and differences between these disorders. Individuals with ASD and primary psychosis activate the same brain regions as healthy controls during facial expression processing, albeit to a different extent. Overall, both groups display altered activation in the fusiform gyrus and amygdala as well as altered connectivity among the broader face processing network, probably indicating reduced facial expression processing abilities. Furthermore, delayed or reduced N170 responses have been reported in ASD and primary psychosis, but the significance of these findings is questioned, and alternative frequency-tagging electroencephalography (EEG) measures are currently explored to capture facial expression processing impairments more selectively. Face perception is an innate process, but it is also guided by visual learning and social experiences. Extreme environmental factors, such as adverse childhood events, can disrupt normative development and alter facial expression processing. ACE are hypothesized to induce altered neural facial expression processing, in particular a hyperactive amygdala response toward negative expressions. Future studies should account for the comorbidity among ASD, primary psychosis, and ACE when assessing facial expression processing in these clinical groups, as it may explain some of the inconsistencies and confound reported in the field.

Rapid neural categorization of angry and fearful faces is specifically impaired in boys with autism spectrum disorder

BACKGROUND

Difficulties with facial expression processing may be associated with the characteristic social impairments in individuals with autism spectrum disorder (ASD). Emotional face processing in ASD has been investigated in an abundance of behavioral and EEG studies, yielding, however, mixed and inconsistent results.

METHODS

We combined fast periodic visual stimulation (FPVS) with EEG to assess the neural sensitivity to implicitly detect briefly presented facial expressions among a stream of neutral faces, in 23 boys with ASD and 23 matched typically developing (TD) boys. Neutral faces with different identities were presented at 6 Hz, periodically interleaved with an expressive face (angry, fearful, happy, sad in separate sequences) every fifth image (i.e., 1.2 Hz oddball frequency). These distinguishable frequency tags for neutral and expressive stimuli allowed direct and objective quantification of the expression-categorization responses, needing only four sequences of 60 s of recording per condition.

RESULTS

Both groups show equal neural synchronization to the general face stimulation and similar neural responses to happy and sad faces. However, the ASD group displays significantly reduced responses to angry and fearful faces, compared to TD boys. At the individual subject level, these neural responses allow to predict membership of the ASD group with an accuracy of 87%. Whereas TD participants show a significantly lower sensitivity to sad faces than to the other expressions, ASD participants show an equally low sensitivity to all the expressions.

CONCLUSIONS

Our results indicate an emotion-specific processing deficit, instead of a general emotion-processing problem: Boys with ASD are less sensitive than TD boys to rapidly and implicitly detect angry and fearful faces. The implicit, fast, and straightforward nature of FPVS-EEG opens new perspectives for clinical diagnosis.

Brain connectivity analysis in fathers of children with autism

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder in which changes in brain connectivity, associated with autistic-like traits in some individuals. First-degree relatives of children with autism may show mild deficits in social interaction. The present study investigates electroencephalography (EEG) brain connectivity patterns of the fathers who have children with autism while performing facial emotion labeling task. Fifteen biological fathers of children with the diagnosis of autism (Test Group) and fifteen fathers of neurotypical children with no personal or family history of autism (Control Group) participated in this study. Facial emotion labeling task was evaluated using a set of photos consisting of six categories (mild and extreme: anger, happiness, and sadness). Group Independent Component Analysis method was applied to EEG data to extract neural sources. Dynamic causal connectivity of neural sources signals was estimated using the multivariate autoregressive model and quantified by using the Granger causality-based methods. Statistical analysis showed significant differences (p value < 0.01) in the connectivity of neural sources in recognition of some emotions in two groups, which the most differences observed in the mild anger and mild sadness emotions. Short-range connectivity appeared in Test Group and conversely, long-range and interhemispheric connections are observed in Control Group. Finally, it can be concluded that the Test Group showed abnormal activity and connectivity in the brain network for the processing of emotional faces compared to the Control Group. We conclude that neural source connectivity analysis in fathers may be considered as a potential and promising biomarker of ASD.

Frequency-Tagging Electroencephalography of Superimposed Social and Non-Social Visual Stimulation Streams Reveals Reduced Saliency of Faces in Autism Spectrum Disorder

Individuals with autism spectrum disorder (ASD) have difficulties with social communication and interaction. The social motivation hypothesis states that a reduced interest in social stimuli may partly underlie these difficulties. Thus far, however, it has been challenging to quantify individual differences in social orientation and interest, and to pinpoint the neural underpinnings of it. In this study, we tested the neural sensitivity for social versus non-social information in 21 boys with ASD (8-12 years old) and 21 typically developing (TD) control boys, matched for age and IQ, while children were engaged in an orthogonal task. We recorded electroencephalography (EEG) during fast periodic visual stimulation (FPVS) of social versus non-social stimuli to obtain an objective implicit neural measure of relative social bias. Streams of variable images of faces and houses were superimposed, and each stream of stimuli was tagged with a particular presentation rate (i.e., 6 and 7.5 Hz or vice versa). This frequency-tagging method allows disentangling the respective neural responses evoked by the different streams of stimuli. Moreover, by using superimposed stimuli, we controlled for possible effects of preferential looking, spatial attention, and disengagement. Based on four trials of 60 s, we observed a significant three-way interaction. In the control group, the frequency-tagged neural responses to faces were larger than those to houses, especially in lateral occipito-temporal channels, while the responses to houses were larger over medial occipital channels. In the ASD group, however, faces and houses did not elicit significantly different neural responses in any of the regions. Given the short recording time of the frequency-tagging paradigm with multiple simultaneous inputs and the robustness of the individual responses, the method could be used as a sensitive marker of social preference in a wide range of populations, including younger and challenging populations.

Fast Periodic Visual Stimulation EEG Reveals Reduced Neural Sensitivity to Fearful Faces in Children with Autism

We objectively quantified the neural sensitivity of school-aged boys with and without autism spectrum disorder (ASD) to detect briefly presented fearful expressions by combining fast periodic visual stimulation with frequency-tagging electroencephalography. Images of neutral faces were presented at 6 Hz, periodically interleaved with fearful expressions at 1.2 Hz oddball rate. While both groups equally display the face inversion effect and mainly rely on information from the mouth to detect fearful expressions, boys with ASD generally show reduced neural responses to rapid changes in expression. At an individual level, fear discrimination responses predict clinical status with an 83% accuracy. This implicit and straightforward approach identifies subtle deficits that remain concealed in behavioral tasks, thereby opening new perspectives for clinical diagnosis.

Happy and Angry Faces Elicit Atypical Neural Activation in Children With Autism Spectrum Disorder

BACKGROUND

Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by significant impairments in social interactions and communication. The ability to accurately perceive and interpret emotional faces is critical to successful social interactions. However, few studies have investigated the spatiotemporal profile of the neural mechanisms underlying emotional face processing in ASD, particularly in children. The current study fills this important gap.

METHODS

Participants were 55 children: 28 children with ASD (mean age = 9.5 ± 1.3 years) and 27 control children (mean age = 8.5 ± 1.3 years). All children completed an implicit emotional face task while magnetoencephalography was recorded. We examined spatiotemporal differences between the groups in neural activation during implicit processing of emotional faces.

RESULTS

Within-group analyses demonstrated greater right middle temporal (300-375 ms) and superior temporal (300-400 ms) activation to angry faces than to happy faces in control children, while children with ASD showed greater activation from 250 to 500 ms to happy faces than to angry faces across frontal and temporal regions. Between-group analyses demonstrated that children with ASD showed similar patterns of late (425-500 ms) posterior cingulate and thalamic underactivity to both angry and happy faces relative to control children, suggesting general atypical processing of emotional information.

CONCLUSIONS

Atypical posterior cingulate cortex and thalamus recruitment in children with ASD to emotional faces suggests poor modulation of toggling between the default mode network and task-based processing. Increased neural activity to happy faces compared with angry faces in children with ASD suggests reduced salience or immature response to anger, which in turn could contribute to deficits in social cognition in ASD.

Reduced neural sensitivity to rapid individual face discrimination in autism spectrum disorder

Background

Individuals with autism spectrum disorder (ASD) are characterized by impairments in social communication and interaction. Although difficulties at processing social signals from the face in ASD have been observed and emphasized for many years, there is a lot of inconsistency across both behavioral and neural studies.

Methods

We recorded scalp electroencephalography (EEG) in 23 8-to-12 year old boys with ASD and 23 matched typically developing boys using a fast periodic visual stimulation (FPVS) paradigm, providing objective (i.e., frequency-tagged), fast (i.e., few minutes) and highly sensitive measures of rapid face categorization, without requiring any explicit face processing task. We tested both the sensitivity to rapidly (i.e., at a glance) categorize faces among other objects and to individuate unfamiliar faces.

Outcomes

While general neural synchronization to the visual stimulation and neural responses indexing generic face categorization were undistinguishable between children with ASD and typically developing controls, neural responses indexing individual face discrimination over the occipito-temporal cortex were substantially reduced in the individuals with ASD. This difference vanished when faces were presented upside-down, due to the lack of significant face inversion effect in ASD.

Interpretation

These data provide original evidence for a selective high-level impairment in individual face discrimination in ASD in an implicit task. The objective and rapid assessment of this function opens new perspectives for ASD diagnosis in clinical settings.

•

We assess implicit face processing in ASD via Fast Periodic Visual Stimulation EEG.

•

Rapid categorization of a face as a face is not impaired in children with ASD.

•

Individual face discrimination is selectively impaired in ASD.

•

Children with ASD show no face inversion effect.

•

FPVS-EEG opens new perspectives for clinical settings.

A Novel Biomarker of Compensatory Recruitment of Face Emotional Imagery Networks in Autism Spectrum Disorder

Imagery of facial expressions in Autism Spectrum Disorder (ASD) is likely impaired but has been very difficult to capture at a neurophysiological level. We developed an approach that allowed to directly link observation of emotional expressions and imagery in ASD, and to derive biomarkers that are able to classify abnormal imagery in ASD. To provide a handle between perception and action imagery cycles it is important to use visual stimuli exploring the dynamical nature of emotion representation. We conducted a case-control study providing a link between both visualization and mental imagery of dynamic facial expressions and investigated source responses to pure face-expression contrasts. We were able to replicate the same highly group discriminative neural signatures during action observation (dynamical face expressions) and imagery, in the precuneus. Larger activation in regions involved in imagery for the ASD group suggests that this effect is compensatory. We conducted a machine learning procedure to automatically identify these group differences, based on the EEG activity during mental imagery of facial expressions. We compared two classifiers and achieved an accuracy of 81% using 15 features (both linear and non-linear) of the signal from theta, high-beta and gamma bands extracted from right-parietal locations (matching the precuneus region), further confirming the findings regarding standard statistical analysis. This robust classification of signals resulting from imagery of dynamical expressions in ASD is surprising because it far and significantly exceeds the good classification already achieved with observation of neutral face expressions (74%). This novel neural correlate of emotional imagery in autism could potentially serve as a clinical interventional target for studies designed to improve facial expression recognition, or at least as an intervention biomarker.