Interactions between facial emotion and identity in face processing: Evidence based on redundancy gains

Dynamic eye avoidance patterns in the high autistic traits group: An eye-tracking study

Introduction

Reduced fixation to the eye area is the main characteristic of social deficits associated with Autism Spectrum Disorder; a similar pattern may exist in individuals with high autistic traits. However, their scanning patterns to the eye area of emotional faces are still unclear on the time scale.

Methods

In the present study, we recruited 46 participants and divided them into the high autistic traits (HAT) group (23 participants) and the low autistic traits (LAT) group (20 participants) based on their Autism Spectrum Quotient (AQ) scores. Moreover, we captured their eye movement patterns when observing different angular emotional faces. We extracted the proportional fixation time to the eye area under different time windows.

Results

The results showed that the fixation time of the HAT group was always significantly smaller than that of the LAT group (p < 0.05), and the difference between the two groups increased in the middle and late stages of face presentation. The results of the linear regression analysis showed that the proportional fixation time was negatively correlated with AQ scores (p < 0.05), indicating that the proportional fixation time to the eye area could be a potential indicator to measure the level of autistic traits. We then calculated the latency to orient the eye area and the latency to disengage the eye area to explore the priority of observation of the eyes. The results showed that compared with the LAT group, the HAT group has a longer latency to orient the eye area (p < 0.05) and has longer latency to disengage the eye area (p < 0.05), illustrating that the HAT group saw the eyes more slowly and left them faster.

The role of discriminability in face perception: Interference processing of expression, gender, and gaze

Eye gaze plays a fundamental role in social interaction and facial recognition. However, interference processing between gaze and other facial variants (e.g., expression) and invariant information (e.g., gender) remains controversial and unclear, especially the role of facial information discriminability in interference. A Garner paradigm was used to conduct two experiments. This paradigm allows simultaneous investigation of the mutual influence of two kinds of facial information in one experiment. In Experiment 1, we manipulated facial expression discriminability and investigated its role in interference processing of gaze and facial expression. The results show that individuals were unable to ignore expression when classifying gaze with both high and low discriminability but could ignore gaze when classifying expression with high discriminability only. In Experiment 2, we manipulated gender discriminability and investigated its function in interference processing of gaze and gender. Participants were unable to ignore gender when classifying gaze with both high and low discriminability but could ignore gaze when classifying gender with low discriminability only. The results indicate that gaze categorization is affected by facial expression and gender regardless of facial information discriminability, whereas interference of gaze on facial expression and gender depends on the degree of discriminability. The present study provides evidence that the processing of gaze and other variant and invariant information is interdependent.

EXPRESS: Attentional Load Effects on Emotional Content in Face Working Memory

This study investigated the role of attentional resources in processing emotional faces in working memory (WM). Participants memorised two face arrays with the same emotion but different identities and were required to judge whether the test face had the same identity as one of the previous faces. Concurrently during encoding and maintenance, a sequence of high-or-low pitched tones (high load) or white noise bursts (low load) was presented, and participants were required to count how many low-tones were heard. Experiments 1 and 2 used an emotional and neutral test face, respectively. Results revealed a significant WM impairment for sad and angry faces in the high load vs low load condition but not for happy faces. In Experiment 1, participants remembered happy faces better than other emotional faces. In contrast, Experiment 2 showed that performance was poorer for happy than sad faces but not for angry faces. This evidence suggests that depleting attentional resources has less impact on WM for happy faces than other emotional faces, but also that differential effects on WM for emotional faces depend on the presence or absence of emotion in the probe face at retrieval.

Integration of Facial Expression and Gaze Direction in Individuals with a High Level of Autistic Traits

BACKGROUND

We investigated whether individuals with high levels of autistic traits integrate relevant communicative signals, such as facial expression, when decoding eye-gaze direction.

METHODS

Students with high vs. low scores on the Autism Spectrum Quotient (AQ) performed a task in which they responded to the eye directions of faces, presented on the left or the right side of a screen, portraying different emotional expressions.

RESULTS

In both groups, the identification of gaze direction was faster when the eyes were directed towards the center of the scene. However, in the low AQ group, this effect was larger for happy faces than for neutral faces or faces showing other emotional expressions, whereas participants from high AQ group were not affected by emotional expressions.

CONCLUSIONS

These results suggest that individuals with more autistic traits may not integrate multiple communicative signals based on their emotional value.

Face-like configurations modulate electrophysiological mismatch responses

The human face is one of the most salient stimuli in the environment. It has been suggested that even basic face-like configurations (three dots composing a downward pointing triangle) may convey salience. Interestingly, stimulus salience can be signaled by mismatch detection phenomena, characterized by greater amplitudes of event-related potentials (ERPs) in response to relevant novel stimulation as compared to non-relevant repeated events. Here, we investigate whether basic face-like stimuli are salient enough to modulate mismatch detection phenomena. ERPs are elicited by a pair of sequentially presented visual stimuli (S1-S2), delivered at a constant 1-s interval, representing either a face-like stimulus (Upright configuration) or three neutral configurations (Inverted, Leftwards, and Rightwards configurations), that are obtained by rotating the Upright configuration along the three different axes. In pairs including a canonical face-like stimulus, we observe a more effective mismatch detection mechanism, with significantly larger N270 and P300 components when S2 is different from S1 as compared to when S2 is identical to S1. This ERP modulation, not significant in pairs excluding face-like stimuli, reveals that mismatch detection phenomena are significantly affected by basic face-like configurations. Even though further experiments are needed to ascertain whether this effect is specifically elicited by face-like configuration rather than by particular orientation changes, our findings suggest that face essential, structural attributes are salient enough to affect change detection processes.

The influence of spatial location on same-different judgments of facial identity and expression

The "spatial congruency bias" is a behavioral phenomenon where 2 objects presented sequentially are more likely to be judged as being the same object if they are presented in the same location (Golomb, Kupitz, & Thiemann, 2014), suggesting that irrelevant spatial location information may be bound to object representations. Here, we examine whether the spatial congruency bias extends to higher-level object judgments of facial identity and expression. On each trial, 2 real-world faces were sequentially presented in variable screen locations, and subjects were asked to make same-different judgments on the facial expression (Experiments 1-2) or facial identity (Experiment 3) of the stimuli. We observed a robust spatial congruency bias for judgments of facial identity, yet a more fragile one for judgments of facial expression. Subjects were more likely to judge 2 faces as displaying the same expression if they were presented in the same location (compared to in different locations), but only when the faces shared the same identity. On the other hand, a spatial congruency bias was found when subjects made judgments on facial identity, even across faces displaying different facial expressions. These findings suggest a possible difference between the binding of facial identity and facial expression to spatial location. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Facial expressions yielding Context-Dependent Effect: The additive contribution of eye movements

This study tested Context-Dependent Effect (CDE) on face recognition by viewing facial expressions as context and face identity as the target. Three groups were defined - Neutral, Happy and Angry, reflecting the facial expressions of the faces presented at the study phase. At the study phase, participants were presented with 42 color photos of faces for 5 s each. At the test phase, participants were presented with 84 pictures of faces, half of which had been viewed beforehand (old). One-third of the old and new faces displayed the same facial expression shown at study, and the remaining two-thirds had one of the other two expressions. Behavioral results show that consistency of facial expressions between study and test facilitated face recognition (i.e., CDE). Eye-tracking results showed that lengthier focus on a face at the study phase gives the participant an advantage only when the same face is presented again at the test phase. Angry expressions intensify binding more than happy or neutral expressions, resulting in higher costs when changing facial expression between study and test. The theoretical implications of these results in terms of the relationship between facial identity and facial expression are discussed. The practical implications, particularly for eyewitness memory literature, are also discussed.

Measuring facial identity and emotion integration using the redundancy gain paradigm

Early theories on face perception posit that invariant (i.e., identity) and changeable (i.e., expression) facial aspects are processed separately. However, many researchers have countered the hypothesis of parallel processes with findings of interactions between identity and emotion perception. The majority of tasks measuring interactions between identity and emotion employ a selective attention design, in which participants are instructed to attend to one dimension (e.g., identity) while the other dimension varies orthogonally (e.g., emotion), but is task irrelevant. Recently, a divided attention design (i.e., the redundancy gain paradigm) in which both identity and emotion are task relevant was employed to assess the interaction between identity and emotion. A redundancy gain is calculated by a drop in reaction time in trials in which a target from both dimensions is present in the stimulus face (e.g., “sad Person A”), compared with trials with only a single target present (e.g., “sad” or “Person A”). Redundancy gains are hypothesized to point to an interactive activation of both dimensions, and as such, could complement designs adopting a selective attention task. The initial aim of the current study was to reproduce the earlier findings with this paradigm on identity and emotion perception (Yankouskaya, Booth, & Humphreys, Attention, Perception, & Psychophysics, 74(8), 1692–1711, 2012), but our study failed to replicate the results. In a series of subtasks, multiple aspects of the design were manipulated separately in our goal to shed light on the factors that influence the redundancy gain effect in faces. A redundancy gain was eventually obtained after controlling for contingencies and stimulus presentation time.

Adults’ Markers of Face Processing Are Present at Age 6 and Are Interconnected Along Development

Recent studies on the development of face processing argue for a late, quantitative, domain-specific development of face processing, and face memory in particular. Most previous findings were based on separately tracking the developmental course of face perception skills, comparing performance across different age groups. Here, we adopted a different approach studying the mechanisms underlying the development of face processing by focusing on how different face skills are interrelated over the years (age 6 to adulthood). Specifically, we examined correlations within and between different categories of tasks: face domain-specific skills involving face recognition based on long-term representations (famous face), and short-term memory retention (Cambridge Face Memory Test), perceptual face-specific marker (inversion effect), global effects in scene perception (global–local task), and the perception of facial expressions. Factor analysis revealed that face identity skills have a similar pattern of interrelations throughout development, identifying two factors: a face domain-specific factor comprising adultlike markers of face processing and a general factor incorporating related, but nonspecific perceptual skills. Domain-specific age-related changes in face recognition entailing short- and long-term retention of face representations were observed, along with mature perceptual face-specific markers and more general perceptual effects predicting face perception skills already at age 6. The results suggest that the domain-specific changes in face processing are unlikely to result from developmental changes in perceptual skills driving face recognition. Instead, development may either involve improvement in the ability to retain face representations in memory or changes in the interactions between the perceptual representations of faces and their representations in long-term memory.

More of me! Distinguishing self and reward bias using redundancy gains

Participants show a perceptual bias favoring stimuli associated with the participants themselves over stimuli associated with other people. A major account of this self-bias effect is that self-related information is intrinsically rewarding, and that high-reward stimuli have enhanced perceptual processing. Here we used redundancy gains to examine the relations between self bias and reward, and whether self and reward biases modulate common levels of stimulus integration. We demonstrated that the self-association bias increases when more than one exemplar of the stimulus is presented (i.e., when participants are exposed to redundant stimuli). The larger self-bias effects for redundant than for single stimuli arose at both perceptual and conceptual levels of representation (respectively, for identical and nonidentical stimuli associated with the same category). In contrast, high-reward stimuli did not affect perceptual redundancy gains with identical shapes, but they did affect redundancy gains with nonidentical stimuli associated with the same category. The strong redundancy effects with self-related stimuli are consistent with self associations modulating stimulus integration at both perceptual and conceptual levels, whereas reward only modulated higher-level conceptual processes (with nonidentical stimuli). The data provide two novel theoretical advances, by showing that (i) self association modulates both early perceptual coding and higher-level conceptual coding, whereas reward only affects the higher-level process, and (ii) self bias can not be reduced simply to differential effects of reward.

What's in a "face file"? Feature binding with facial identity, emotion, and gaze direction

A series of four experiments investigated the binding of facial (i.e., facial identity, emotion, and gaze direction) and non-facial (i.e., spatial location and response location) attributes. Evidence for the creation and retrieval of temporary memory face structures across perception and action has been adduced. These episodic structures-dubbed herein "face files"-consisted of both visuo-visuo and visuo-motor bindings. Feature binding was indicated by partial-repetition costs. That is repeating a combination of facial features or altering them altogether, led to faster responses than repeating or alternating only one of the features. Taken together, the results indicate that: (a) "face files" affect both action and perception mechanisms, (b) binding can take place with facial dimensions and is not restricted to low-level features (Hommel, Visual Cognition 5:183-216, 1998), and (c) the binding of facial and non-facial attributes is facilitated if the dimensions share common spatial or motor codes. The theoretical contributions of these results to "person construal" theories (Freeman, & Ambady, Psychological Science, 20(10), 1183-1188, 2011), as well as to face recognition models (Haxby, Hoffman, & Gobbini, Biological Psychiatry, 51(1), 59-67, 2000) are discussed.

Expanding and retracting from the self: Gains and costs in switching self-associations

We report 2 experiments to assess the strength of forming and breaking associations to the self, familiar others, and unfamiliar others in a simple shape–label matching task. In each experiment, participants first formed shape–person associations (e.g., triangle-self). Subsequently, they had to relearn the associations with the shapes and labels rearranged (self→stranger in Experiment 1; self→friend in Experiment 2) and they carried out a matching task in which they judged whether shape–label stimuli were as newly instructed or re-paired. There were faster responses and fewer errors on match trials for newly formed self-associated stimuli. In contrast, after switching, reaction times were slower and accuracy was reduced on mismatch trials involving shapes previously associated with the self. The strength of the self-advantage in forming the new association on match trials correlated with the difficulty in switching from the old self-associated shape on mismatch trials. The results indicate that self-reference enhances the binding of associations in memory; this facilitates associations to new stimuli, but there is a cost of interference from old associations.

General recognition theory with individual differences: a new method for examining perceptual and decisional interactions with an application to face perception

A common question in perceptual science is to what extent different stimulus dimensions are processed independently. General recognition theory (GRT) offers a formal framework via which different notions of independence can be defined and tested rigorously, while also dissociating perceptual from decisional factors. This article presents a new GRT model that overcomes several shortcomings with previous approaches, including a clearer separation between perceptual and decisional processes and a more complete description of such processes. The model assumes that different individuals share similar perceptual representations, but vary in their attention to dimensions and in the decisional strategies they use. We apply the model to the analysis of interactions between identity and emotional expression during face recognition. The results of previous research aimed at this problem have been disparate. Participants identified four faces, which resulted from the combination of two identities and two expressions. An analysis using the new GRT model showed a complex pattern of dimensional interactions. The perception of emotional expression was not affected by changes in identity, but the perception of identity was affected by changes in emotional expression. There were violations of decisional separability of expression from identity and of identity from expression, with the former being more consistent across participants than the latter. One explanation for the disparate results in the literature is that decisional strategies may have varied across studies and influenced the results of tests of perceptual interactions, as previous studies lacked the ability to dissociate between perceptual and decisional interactions.

The processing of facial identity and expression is interactive, but dependent on task and experience

Facial identity and emotional expression are two important sources of information for daily social interaction. However the link between these two aspects of face processing has been the focus of an unresolved debate for the past three decades. Three views have been advocated: (1) separate and parallel processing of identity and emotional expression signals derived from faces; (2) asymmetric processing with the computation of emotion in faces depending on facial identity coding but not vice versa; and (3) integrated processing of facial identity and emotion. We present studies with healthy participants that primarily apply methods from mathematical psychology, formally testing the relations between the processing of facial identity and emotion. Specifically, we focused on the "Garner" paradigm, the composite face effect and the divided attention tasks. We further ask whether the architecture of face-related processes is fixed or flexible and whether (and how) it can be shaped by experience. We conclude that formal methods of testing the relations between processes show that the processing of facial identity and expressions interact, and hence are not fully independent. We further demonstrate that the architecture of the relations depends on experience; where experience leads to higher degree of inter-dependence in the processing of identity and expressions. We propose that this change occurs as integrative processes are more efficient than parallel. Finally, we argue that the dynamic aspects of face processing need to be incorporated into theories in this field.

Interactions between Identity and Emotional Expression in Face Processing across the Lifespan: Evidence from Redundancy Gains

We tested how aging affects the integration of visual information from faces. Three groups of participants aged 20–30, 40–50, and 60–70 performed a divided attention task in which they had to detect the presence of a target facial identity or a target facial expression. Three target stimuli were used: (1) with the target identity but not the target expression, (2) with the target expression but not the target identity, and (3) with both the target identity and target expression (the redundant target condition). On nontarget trials the faces contained neither the target identity nor expression. All groups were faster in responding to a face containing both the target identity and emotion compared to faces containing either single target. Furthermore the redundancy gains for combined targets exceeded performance limits predicted by the independent processing of facial identity and emotion. These results are held across the age range. The results suggest that there is interactive processing of facial identity and emotion which is independent of the effects of cognitive aging. Older participants demonstrated reliably larger size of the redundancy gains compared to the young group that reflect a greater experience with faces. Alternative explanations are discussed.