Beyond face value: does involuntary emotional anticipation shape the perception of dynamic facial expressions?

Visual Attention to Dynamic Emotional Faces in Adults on the Autism Spectrum

Using eye-tracking, we studied allocation of attention to faces where the emotional expression and eye-gaze dynamically changed in an ecologically-valid manner. We tested typically-developed (TD) adults low or high in autistic-like traits (Experiment 1), and adults with high-functioning autism (HFA; Experiment 2). All groups fixated more on the eyes than on any of the other facial area, regardless of emotion and gaze direction, though the HFA group fixated less on the eyes and more on the nose than TD controls. The sequence of dynamic facial changes affected the groups similarly, with reduced attention to the eyes and increased attention to the mouth. The results suggest that dynamic emotional face scanning patterns are stereotypical and differ only modestly between TD and HFA adults.

Emotional anticipation for dynamic emotional faces is not modulated by schizotypal traits: A Representational Momentum study

Schizotypy, a personality structure that resembles schizophrenia symptoms, is often associated with abnormal facial emotion perception. Based on the prevailing sense of threat in psychotic experiences, and the immediate perceptual history of seeing others' facial expressions, individuals with high schizotypal traits may exhibit a heightened tendency to anticipate anger. To test this, we used insights from Representational Momentum (RM), a perceptual phenomenon in which the endpoint of a dynamic event is systematically displaced forward, into the immediate future. Angry-to-ambiguous and happy-to-ambiguous avatar faces were presented, each followed by a probe with the same (ambiguous) expression as the endpoint, or one slightly changed to express greater happiness/anger. Participants judged if the probe was "equal" to the endpoint and rated how confident they were. The sample was divided into high (N = 46) and low (N = 49) schizotypal traits using the Schizotypal Personality Questionnaire (SPQ). First, a forward bias was found in happy-to-ambiguous faces, suggesting emotional anticipation solely for dynamic faces changing towards a potential threat (anger). This may reflect an adaptative mechanism, as it is safer to anticipate any hostility from a conspecific than the opposite. Second, contrary to our hypothesis, high schizotypal traits did not heighten RM for happy-to-ambiguous faces, nor did they lead to overconfidence in biased judgements. This may suggest a typical pattern of emotional anticipation in non-clinical schizotypy, but caution is needed due to the use of self-report questionnaires, university students, and a modest sample size. Future studies should also investigate if the same holds for clinical manifestations of schizophrenia.

Are adaptation aftereffects for facial emotional expressions affected by prior knowledge about the emotion?

Accurate perception of the emotional signals conveyed by others is crucial for successful social interaction. Such perception is influenced not only by sensory input, but also by knowledge we have about the others' emotions. This study addresses the issue of whether knowing that the other's emotional state is congruent or incongruent with their displayed emotional expression ("genuine" and "fake", respectively) affects the neural mechanisms underpinning the perception of their facial emotional expressions. We used a visual adaptation paradigm to investigate this question in three experiments employing increasing adaptation durations. The adapting stimuli consisted of photographs of emotional facial expressions of joy and anger, purported to reflect (in-)congruency between felt and expressed emotion, displayed by professional actors. A Validity checking procedure ensured participants had the correct knowledge about the (in-)congruency. Significantly smaller adaptation aftereffects were obtained when participants knew that the displayed expression was incongruent with the felt emotion, following all tested adaptation periods. This study shows that knowledge relating to the congruency between felt and expressed emotion modulates face expression aftereffects. We argue that this reflects that the neural substrate responsible for the perception of facial expressions of emotion incorporates the presumed felt emotion underpinning the expression.

Stria terminalis, amygdala, and temporoparietal junction networks facilitate efficient emotion processing under expectations

Rapid emotion processing is an ecologically essential ability for survival in social environments in which threatening or advantageous encounters dynamically and rapidly occur. Efficient emotion recognition is subserved by different processes, depending on one's expectations; however, the underlying functional and structural circuitry is still poorly understood. In this study, we delineate brain networks that subserve fast recognition of emotion in situations either congruent or incongruent with prior expectations. For this purpose, we used multimodal neuroimaging and investigated performance on a dynamic emotion perception task. We show that the extended amygdala structural and functional networks relate to speed of emotion processing under threatening conditions. Specifically, increased microstructure of the right stria terminalis, an amygdala white-matter pathway, was related to faster detection of emotion during actual presentation of anger or after cueing anger. Moreover, functional connectivity of right amygdala with limbic regions was related to faster detection of anger congruent with cue, suggesting selective attention to threat. On the contrary, we found that faster detection of anger incongruent with cue engaged the ventral attention "reorienting" network. Faster detection of happiness, in either expectancy context, engaged a widespread frontotemporal-subcortical functional network. These findings shed light on the functional and structural circuitries that facilitate speed of emotion recognition and, for the first time, elucidate a role for the stria terminalis in human emotion processing.

Sensorimotor cortex activation during anticipation of upcoming predictable but not unpredictable actions

The mirror neuron system (MNS) becomes active during action execution and action observation, which is presumably reflected by reductions in mu (8-13 Hz) activity in the electroencephalogram over the sensorimotor cortex. The function of the MNS is still fiercely debated. The current study aimed to investigate a role of the MNS in anticipating others' actions by examining whether the MNS was activated - indexed by mu power suppression - prior to the onset of observed actions when the onset and type of action could be predicted on the basis of environmental cues. Young adults performed and observed cued grasping and placing actions in a card game in a real-life setting, while the predictability of the observed actions was manipulated using rules. Significant mu suppression, relative to within-trial baseline activity, was found both prior to and during executed actions, but also during action observation, and, crucially, prior to observed actions provided they were predictable. No anticipatory mu reductions were found prior to unpredictable observed actions. These results suggest top-down modulation of MNS activity by conceptual knowledge. This is the first study to demonstrate mu suppression prior to action onset - possibly reflecting MNS anticipatory activity - by explicitly manipulating predictability.

The Role of Pattern Extrapolation in the Perception of Dynamic Facial Expressions in Autism Spectrum Disorder

Changes in the intensity and type of facial expressions reflect alterations in the emotional state of the agent. Such "direct" access to the other's affective state might, top-down, influence the perception of the facial expressions that gave rise to the affective state inference. Previously, we described a perceptual bias occurring when the last, neutral, expression of offsets of facial expressions (joy-to-neutral and anger-to-neutral), was evaluated. Individuals with high-functioning autism (HFA) and matched typically developed (TD) individuals rated the neutral expression at the end of the joy-offset videos as slightly angry and the identical neutral expression at the end of the anger-offset videos as slightly happy ("overshoot" bias). That study suggested that the perceptual overshoot response bias in the TD group could be best explained by top-down "emotional anticipation," i.e., the involuntary/automatic anticipation of the agent's next emotional state of mind, generated by the immediately preceding perceptual history (low-level mind reading). The experimental manipulations further indicated that in the HFA group the "overshoot" was better explained by contrast effects between the first and last facial expressions, both presented for a relatively long period of 400 ms. However, in principle, there is another, more parsimonious, explanation, which is pattern extrapolation or representational momentum (RM): the extrapolation of a pattern present in the dynamic sequence. This hypothesis is tested in the current study, in which 18 individuals with HFA and a matched control group took part. In a base-line condition, joy-offset and anger-offset video-clips were presented. In the new experimental condition, the clips were modified so as to create an offset-onset-offset pattern within each sequence (joy-to-anger-to-neutral and anger-to-joy-to-neutral). The final neutral expressions had to be evaluated. The overshoot bias was confirmed in the base-line condition for both TD and HFA groups, while the experimental manipulation removed the bias in both groups. This outcome ruled out pattern extrapolation or RM as explanation for the perceptual "overshoot" bias in the HFA group and suggested a role for facial contrast effects in HFA. This is compatible with the view that ASD individuals tend to lack the spontaneous "tracking" of changes in the others' affective state and hence show no or reduced emotional anticipation.

The neuroelectric dynamics of the emotional anticipation of other people's pain

When we observe a dynamic emotional facial expression, we usually automatically anticipate how that expression will develop. Our objective was to study a neurocognitive biomarker of this anticipatory process for facial pain expressions, operationalized as a mismatch effect. For this purpose, we studied the behavioral and neuroelectric (Event-Related Potential, ERP) correlates, of a match or mismatch, between the intensity of an expression of pain anticipated by the participant, and the intensity of a static test expression of pain displayed with the use of a representational momentum paradigm. Here, the paradigm consisted in displaying a dynamic facial pain expression which suddenly disappeared, and participants had to memorize the final intensity of the dynamic expression. We compared ERPs in response to congruent (intensity the same as the one memorized) and incongruent (intensity different from the one memorized) static expression intensities displayed after the dynamic expression. This paradigm allowed us to determine the amplitude and direction of this intensity anticipation by measuring the observer's memory bias. Results behaviorally showed that the anticipation was backward (negative memory bias) for high intensity expressions of pain (participants expected a return to a neutral state) and more forward (memory bias less negative, or even positive) for less intense expressions (participants expected increased intensity). Detecting mismatch (incongruent intensity) led to faster responses than detecting match (congruent intensity). The neuroelectric correlates of this mismatch effect in response to the testing of expression intensity ranged from P100 to LPP (Late Positive Potential). Path analysis and source localization suggested that the medial frontal gyrus was instrumental in mediating the mismatch effect through top-down influence on both the occipital and temporal regions. Moreover, having the facility to detect incongruent expressions, by anticipating emotional state, could be useful for prosocial behavior and the detection of trustworthiness.

Association between schizophrenia polygenic risk and neural correlates of emotion perception

The neural correlates of emotion perception have been shown to be significantly altered in schizophrenia (SCZ) patients as well as their healthy relatives, possibly reflecting genetic susceptibility to the disease. The aim of the study was to investigate the association between SCZ polygenic risk and brain activity whilst testing perception of multisensory, dynamic emotional stimuli. We created SCZ polygenic risk scores (PRS) for a sample of twenty-eight healthy individuals. The PRS was based on data from the Psychiatric Genomics Consortium and was used as a regressor score in the neuroimaging analysis. The results of a multivariate brain-behaviour analysis show that higher SCZ PRS are related to increased activity in brain regions critical for emotion during the perception of threatening (angry) emotions. These results suggest that individuals with higher SCZ PRS over-activate the neural correlates underlying emotion during perception of threat, perhaps due to an increased experience of fear or neural inefficiency in emotion-regulation areas. Moreover, over-recruitment of emotion regulation regions might function as a compensation to maintain normal emotion regulation during threat perception. If replicated in larger studies, these findings may have important implications for understanding the neurophysiological biomarkers relevant in SCZ.

Context sensitivity in the detection of changes in facial emotion

In social contexts, reading subtle changes in others’ facial expressions is a crucial communication skill. To measure this ability, we developed an expression-change detection task, wherein a series of pictures of changes in an individual’s facial expressions within contextual scenes were presented. The results demonstrated that the detection of subtle changes was highly sensitive to contextual stimuli. That is, participants identified the direction of facial-expression changes more accurately and more quickly when they were ‘appropriate’—consistent with the valence of the contextual stimulus change—than when they were ‘inappropriate’. Moreover, individual differences in sensitivity to contextual stimuli were correlated with scores on the Toronto Alexithymia Scale, a commonly used measure of alexithymia tendencies. These results suggest that the current behavioural task might facilitate investigations of the role of context in human social cognition.

Dynamic emotion perception and prior expectancy

Social interactions require the ability to rapidly perceive emotion from various incoming dynamic, multisensory cues. Prior expectations reduce incoming emotional information and direct attention to cues that are aligned with what is expected. Studies to date have investigated the prior expectancy effect using static emotional images, despite the fact that dynamic stimuli would represent greater ecological validity. The objective of the study was to create a novel functional magnetic resonance imaging (fMRI) paradigm to examine the influence of prior expectations on naturalistic emotion perception. For this purpose, we developed a dynamic emotion perception task, which consisted of audio-visual videos that carry emotional information congruent or incongruent with prior expectations. The results show that emotional congruency was associated with activity in prefrontal regions, amygdala, and putamen, whereas emotional incongruency was associated with activity in temporoparietal junction and mid-cingulate gyrus. Supported by the behavioural results, our findings suggest that prior expectations are reinforced after repeated experience and learning, whereas unexpected emotions may rely on fast change detection processes. The results from the current study are compatible with the notion that the ability to automatically detect unexpected changes in complex dynamic environments allows for adaptive behaviours in potentially advantageous or threatening situations.

Fashioning the Face: Sensorimotor Simulation Contributes to Facial Expression Recognition

When we observe a facial expression of emotion, we often mimic it. This automatic mimicry reflects underlying sensorimotor simulation that supports accurate emotion recognition. Why this is so is becoming more obvious: emotions are patterns of expressive, behavioral, physiological, and subjective feeling responses. Activation of one component can therefore automatically activate other components. When people simulate a perceived facial expression, they partially activate the corresponding emotional state in themselves, which provides a basis for inferring the underlying emotion of the expresser. We integrate recent evidence in favor of a role for sensorimotor simulation in emotion recognition. We then connect this account to a domain-general understanding of how sensory information from multiple modalities is integrated to generate perceptual predictions in the brain.

Neuromodulation, Emotional Feelings and Affective Disorders

Affective disorders such as anxiety, phobia and depression are a leading cause of disabilities worldwide. Monoamine neuromodulators are used to treat most of them, with variable degrees of efficacy. Here, we review and interpret experimental findings about the relation of neuromodulation and emotional feelings, in pursuit of two goals: (a) to improve the conceptualisation of affective/emotional states, and (b) to develop a descriptive model of basic emotional feelings related to the actions of neuromodulators. In this model, we hypothesize that specific neuromodulators are effective for basic emotions. The model can be helpful for mental health professionals to better understand the affective dynamics of persons and the actions of neuromodulators - and respective psychoactive drugs - on this dynamics.

Atypical emotional anticipation in high-functioning autism

BACKGROUND

Understanding and anticipating others' mental or emotional states relies on the processing of social cues, such as dynamic facial expressions. Individuals with high-functioning autism (HFA) may process these cues differently from individuals with typical development (TD) and purportedly use a 'mechanistic' rather than a 'mentalistic' approach, involving rule- and contingency-based interpretations of the stimuli. The study primarily aimed at examining whether the judgments of facial expressions made by individuals with TD and HFA would be similarly affected by the immediately preceding dynamic perceptual history of that face. A second aim was to explore possible differences in the mechanisms underpinning the perceptual judgments in the two groups.

METHODS

Twenty-two adults with HFA and with TD, matched for age, gender and IQ, were tested in three experiments in which dynamic, 'ecologically valid' offsets of happy and angry facial expressions were presented. Participants evaluated the expression depicted in the last frame of the video clip by using a 5-point scale ranging from slightly angry via neutral to slightly happy. Specific experimental manipulations prior to the final facial expression of the video clip allowed examining contributions of bottom-up mechanisms (sequential contrast/context effects and representational momentum) and a top-down mechanism (emotional anticipation) to distortions in the perception of the final expression.

RESULTS

In experiment 1, the two groups showed a very similar perceptual bias for the final expression of joy-to-neutral and anger-to-neutral videos (overshoot bias). In experiment 2, a change in the actor's identity during the clip removed the bias in the TD group, but not in the HFA group. In experiment 3, neutral-to-joy/anger-to-neutral sequences generated an undershoot bias (opposite to the overshoot) in the TD group, whereas no bias was observed in the HFA group.

CONCLUSIONS

We argue that in TD individuals the perceptual judgments of other's facial expressions were underpinned by an automatic emotional anticipation mechanism. In contrast, HFA individuals were primarily influenced by visual features, most notably the contrast between the start and end expressions, or pattern extrapolation. We critically discuss the proposition that automatic emotional anticipation may be induced by motor simulation of the perceived dynamic facial expressions and discuss its implications for autism.

The new hybrids: Continuing debates on social perception

I evaluate several attempts to integrate standard theories of social cognition, either theory theory or simulation theory, with aspects of interaction theory, and especially with the concept of direct social perception. I refer to these as new hybrid theories of social cognition. One of the new hybrids accomplishes the integration only by weakening the concept of mindreading or by understanding mindreading as targeting the shared situation rather than the other's mental states. Hybrids that attempt to accommodate the idea of direct perception of mental states grant a phenomenological directness only by maintaining tacit (theory-based) inferences on the subpersonal level. If such inferential processes are thought to be extra-perceptual, then perception is neither sufficient nor direct for an understanding of intentions and emotions. Moreover, insistence on top-down inferential processes trades off against the possibility of plasticity in the perceptual system itself. I suggest that a better model than a hybrid theory would be a pluralist one. A pluralist approach to social cognition would treat theoretical inference, simulation, direct perception, interactive skills, etc. as different strategies. The real challenge is to work out a pluralist account of subpersonal processes.

The direct perception hypothesis: perceiving the intention of another's action hinders its precise imitation

We argue that imitation is a learning response to unintelligible actions, especially to social conventions. Various strands of evidence are converging on this conclusion, but further progress has been hampered by an outdated theory of perceptual experience. Comparative psychology continues to be premised on the doctrine that humans and non-human primates only perceive others' physical "surface behavior," while mental states are perceptually inaccessible. However, a growing consensus in social cognition research accepts the direct perception hypothesis: primarily we see what others aim to do; we do not infer it from their motions. Indeed, physical details are overlooked - unless the action is unintelligible. On this basis we hypothesize that apes' propensity to copy the goal of an action, rather than its precise means, is largely dependent on its perceived intelligibility. Conversely, children copy means more often than adults and apes because, uniquely, much adult human behavior is completely unintelligible to unenculturated observers due to the pervasiveness of arbitrary social conventions, as exemplified by customs, rituals, and languages. We expect the propensity to imitate to be inversely correlated with the familiarity of cultural practices, as indexed by age and/or socio-cultural competence. The direct perception hypothesis thereby helps to parsimoniously explain the most important findings of imitation research, including children's over-imitation and other species-typical and age-related variations.