Turning the other cheek: the viewpoint dependence of facial expression after-effects

Adaptation aftereffects reveal how categorization training changes the encoding of face identity

Previous research suggests that learning to categorize faces along a novel dimension changes the perceptual representation of such dimension, increasing its discriminability, its invariance, and the information used to identify faces varying along the dimension. A common interpretation of these results is that categorization training promotes the creation of novel dimensions, rather than simply the enhancement of already existing representations. Here, we trained a group of participants to categorize faces that varied along two morphing dimensions, one of them relevant to the categorization task and the other irrelevant to the task. An untrained group did not receive such categorization training. In three experiments, we used face adaptation aftereffects to explore how categorization training changes the encoding of face identities at the extremes of the category-relevant dimension and whether such training produces encoding of the category-relevant dimension as a preferred direction in face space. The pattern of results suggests that categorization training enhances the already existing norm-based coding of face identity, rather than creating novel category-relevant representations. We formalized this conclusion in a model that explains the most important results in our experiments and serves as a working hypothesis for future work in this area.

Asymmetric neural responses for facial expressions and anti-expressions

Face recognition requires identifying both the invariant characteristics that distinguish one individual from another and the variations within the individual that correspond to emotional expressions. Both have been postulated to be represented via a norm-based code, in which identity or expression are represented as deviations from an average or neutral prototype. We used Fast Periodic Visual Stimulation (FPVS) with electroencephalography (EEG) to compare neural responses for neutral faces, expressions and anti-expressions. Anti-expressions are created by projecting an expression (e.g. a happy face) through the neutral face to form the opposite facial shape (anti-happy). Thus expressions and anti-expressions differ from the norm by the same "configural" amount and thus have equivalent but opposite status with regard to their shape, but differ in their ecological validity. We examined whether neural responses to these complementary stimulus pairs were equivalent or asymmetric, and also tested for norm-based coding by comparing whether stronger responses are elicited by expressions and anti-expressions than neutral faces. Observers viewed 20 s sequences of 6 Hz alternations of neutral faces and expressions, neutral faces and anti-expressions, and expressions and anti-expressions. Responses were analyzed in the frequency domain. Significant responses at half the frequency of the presentation rate (3 Hz), indicating asymmetries in responses, were observed for all conditions. Inversion of the images reduced the size of this signal, indicating these asymmetries are not solely due to differences in the low-level properties of the images. While our results do not preclude a norm-based code for expressions, similar to identity, this representation (as measured by the FPVS EEG responses) may also include components sensitive to which configural distortions form meaningful expressions.

Intensity dependence in high-level facial expression adaptation aftereffect

Perception of a facial expression can be altered or biased by a prolonged viewing of other facial expressions, known as the facial expression adaptation aftereffect (FEAA). Recent studies using antiexpressions have demonstrated a monotonic relation between the magnitude of the FEAA and adaptor extremity, suggesting that facial expressions are opponent coded and represented continuously from one expression to its antiexpression. However, it is unclear whether the opponent-coding scheme can account for the FEAA between two facial expressions. In the current study, we demonstrated that the magnitude of the FEAA between two facial expressions increased monotonically as a function of the intensity of adapting facial expressions, consistent with the predictions based on the opponent-coding model. Further, the monotonic increase in the FEAA occurred even when the intensity of an adapting face was too weak for its expression to be recognized. These results together suggest that multiple facial expressions are encoded and represented by balanced activity of neural populations tuned to different facial expressions.

Perceived Duration Increases with Contrast, but Only a Little

Recent adaptation studies provide evidence for early visual areas playing a role in duration perception. One explanation for the pronounced duration compression commonly found with adaptation is that it reflects adaptation-driven stimulus-specific reduction in neural activity in early visual areas. If this level of stimulus-associated neural activity does drive duration, then we would expect a strong effect of contrast on perceived duration as electrophysiological studies shows neural activity in early visual areas to be strongly related to contrast. We employed a spatially isotropic noise stimulus where the luminance of each noise element was independently sinusoidally modulated at 4 Hz. Participants matched the perceived duration of a high (0.9) or low (0.1) contrast stimulus to a previously presented standard stimulus (600 ms, contrast = 0.3). To achieve perceptually equivalent durations, the low contrast stimulus had to be presented for longer than the high contrast stimulus. This occurred when we controlled for stimulus size and when we adjusted for individual differences in perceived temporal frequency. Further, we show that the effect cannot be explained by shifts in perceived onset and offset and is not explained by a simple contrast-driven response bias. The direction of our results is clearly consistent with the idea that level of neural activity drives duration. However, the magnitude of the effect (~10% duration difference over a 0.9-0.1 contrast reduction) is in marked contrast to the larger duration distortions that can be found with repetition suppression and the oddball effect; particularly when these may be associated with smaller differences in neural activity than that expected from our contrast difference. Taken together, these results indicate that level of stimulus-related neural activity in early visual areas is unlikely to provide a general mechanism for explaining differences in perceived duration.

Naltrexone alters the processing of social and emotional stimuli in healthy adults

Endogenous opioids have complex social effects that may depend on specific receptor actions and vary depending on the "stage" of social behavior (e.g., seeking vs. responding to social stimuli). We tested the effects of a nonspecific opioid antagonist, naltrexone (NTX), on social processing in humans. NTX is used to treat alcohol and opiate dependence, and may affect both mu and kappa-opioid systems. We assessed attention ("seeking"), and subjective and psychophysiological responses ("responding") to positive and negative social stimuli. Based on literature suggesting mu-opioid blockade impairs positive social responses, we hypothesized that NTX would decrease responses to positive social stimuli. We also tested responses to negative stimuli, which might be either increased by NTX's mu-opioid effects or decreased by its kappa-opioid effects. Thirty-four healthy volunteers received placebo, 25 mg, or 50 mg NTX across three sessions under double-blind conditions. At each session, participants completed measures of attention, identification, and emotional responses for emotional faces and scenes. NTX increased attention to emotional expressions, slowed identification of sadness and fear, and decreased ratings of arousal for social and nonsocial emotional scenes. These findings are more consistent with anxiolytic kappa-antagonist than mu-blocking effects, suggesting effects on kappa receptors may contribute to the clinical effects of NTX.

MDMA alters emotional processing and facilitates positive social interaction

BACKGROUND

±3,4-Methylenedioxymethamphetamine (MDMA, "ecstasy") produces "prosocial" effects, such as feelings of empathy and closeness, thought to be important to its abuse and its value in psychotherapy. However, it is not fully understood how MDMA alters basic emotional processes to produce these effects, or whether it produces corresponding changes in actual social behavior. Here, we examined how MDMA affects perceptions of and responses to emotional expressions, and tested its effects on behavior during a social interaction. We also examined whether MDMA's prosocial effects related to a measure of abuse liability.

METHODS

Over three sessions, 36 healthy volunteers with previous ecstasy use received MDMA (0.75, 1.5 mg/kg) and placebo under double-blind conditions. We measured (i) mood and cardiovascular effects, (ii) perception of and psychophysiological responses to emotional expressions, (iii) use of positive and negative words in a social interaction, and (iv) perceptions of an interaction partner. We then tested whether these effects predicted desire to take the drug again.

RESULTS

MDMA slowed perception of angry expressions, increased psychophysiological responses to happy expressions, and increased positive word use and perceptions of partner empathy and regard in a social interaction. These effects were not strongly related to desire to take the drug again.

CONCLUSIONS

MDMA alters basic emotional processes by slowing identification of negative emotions and increasing responses to positive emotions in others. Further, it positively affects behavior and perceptions during actual social interaction. These effects may contribute to the efficacy of MDMA in psychotherapy, but appear less closely related to its abuse potential.

Looking at a Blurry Old Family Photo? Zoom Out!

We investigated recognition of blurry faces and whether viewing size affects identification of such severely degraded images. Despite the common belief that face perception relies on middle spatial frequencies, the critical spatial frequency band for face recognition is not fixed but rather depends on size. This is especially pronounced at small sizes, where observers choose to utilize lower, rather than middle, frequencies to identify a face. Here we assessed recognition of identity via a novel use of the face adaptation paradigm. We examined face identity aftereffects of blurry and intact adaptors at two sizes. Intact adaptors induced significant aftereffects regardless of size. Small, but not large, blurry adaptors produced aftereffects despite the fact that both contained exactly the same level of facial detail. This suggests an inability to utilize low-frequency information for perceiving identity in large faces. We conclude that (1) size is a key factor in human face recognition processes and (2) coarse facial images are better recognized at small sizes.

Residual fMRI sensitivity for identity changes in acquired prosopagnosia

While a network of cortical regions contribute to face processing, the lesions in acquired prosopagnosia are highly variable, and likely result in different combinations of spared and affected regions of this network. To assess the residual functional sensitivities of spared regions in prosopagnosia, we designed a rapid event-related functional magnetic resonance imaging (fMRI) experiment that included pairs of faces with same or different identities and same or different expressions. By measuring the release from adaptation to these facial changes we determined the residual sensitivity of face-selective regions-of-interest. We tested three patients with acquired prosopagnosia, and all three of these patients demonstrated residual sensitivity for facial identity changes in surviving fusiform and occipital face areas of either the right or left hemisphere, but not in the right posterior superior temporal sulcus. The patients also showed some residual capabilities for facial discrimination with normal performance on the Benton Facial Recognition Test, but impaired performance on more complex tasks of facial discrimination. We conclude that fMRI can demonstrate residual processing of facial identity in acquired prosopagnosia, that this adaptation can occur in the same structures that show similar processing in healthy subjects, and further, that this adaptation may be related to behavioral indices of face perception.

Amphetamine as a social drug: effects of d-amphetamine on social processing and behavior

RATIONALE

Drug users often report using drugs to enhance social situations, and empirical studies support the idea that drugs increase both social behavior and the value of social interactions. One way that drugs may affect social behavior is by altering social processing, for example by decreasing perceptions of negative emotion in others.

OBJECTIVES

We examined effects of d-amphetamine on processing of emotional facial expressions and on the social behavior of talking. We predicted amphetamine would enhance attention, identification, and responsivity to positive expressions, and that this in turn would predict increased talkativeness.

METHODS

Over three sessions, 36 healthy normal adults received placebo, 10, and 20 mg d-amphetamine under counterbalanced double-blind conditions. At each session, we measured processing of happy, fearful, sad, and angry expressions using an attentional visual probe task, a dynamic emotion identification task, and measures of facial muscle activity. We also measured talking.

RESULTS

Amphetamine decreased the threshold for identifying all emotions, increased negative facial responses to sad expressions, and increased talkativeness. Contrary to our hypotheses, amphetamine did not alter attention to, identification of, or facial responses to positive emotions specifically. Interestingly, the drug decreased the threshold to identify all emotions, and this effect was uniquely related to increased talkativeness, even after controlling for overall sensitivity to amphetamine.

CONCLUSIONS

The results suggest that amphetamine may encourage sociability by increasing sensitivity to subtle emotional expressions. These findings suggest novel social mechanisms that may contribute to the rewarding effects of amphetamine.

The MPI facial expression database--a validated database of emotional and conversational facial expressions

The ability to communicate is one of the core aspects of human life. For this, we use not only verbal but also nonverbal signals of remarkable complexity. Among the latter, facial expressions belong to the most important information channels. Despite the large variety of facial expressions we use in daily life, research on facial expressions has so far mostly focused on the emotional aspect. Consequently, most databases of facial expressions available to the research community also include only emotional expressions, neglecting the largely unexplored aspect of conversational expressions. To fill this gap, we present the MPI facial expression database, which contains a large variety of natural emotional and conversational expressions. The database contains 55 different facial expressions performed by 19 German participants. Expressions were elicited with the help of a method-acting protocol, which guarantees both well-defined and natural facial expressions. The method-acting protocol was based on every-day scenarios, which are used to define the necessary context information for each expression. All facial expressions are available in three repetitions, in two intensities, as well as from three different camera angles. A detailed frame annotation is provided, from which a dynamic and a static version of the database have been created. In addition to describing the database in detail, we also present the results of an experiment with two conditions that serve to validate the context scenarios as well as the naturalness and recognizability of the video sequences. Our results provide clear evidence that conversational expressions can be recognized surprisingly well from visual information alone. The MPI facial expression database will enable researchers from different research fields (including the perceptual and cognitive sciences, but also affective computing, as well as computer vision) to investigate the processing of a wider range of natural facial expressions.

Shared or separate mechanisms for self-face and other-face processing? Evidence from adaptation

Evidence that self-face recognition is dissociable from general face recognition has important implications both for models of social cognition and for our understanding of face recognition. In two studies, we examine how adaptation affects the perception of personally familiar faces, and we use a visual adaptation paradigm to investigate whether the neural mechanisms underlying the recognition of one’s own and other faces are shared or separate. In Study 1 we show that the representation of personally familiar faces is rapidly updated by visual experience with unfamiliar faces, so that the perception of one’s own face and a friend’s face is altered by a brief period of adaptation to distorted unfamiliar faces. In Study 2, participants adapted to images of their own and a friend’s face distorted in opposite directions; the contingent aftereffects we observe are indicative of separate neural populations, but we suggest that these reflect coding of facial identity rather than of the categories “self” and “other.”

From single cells to social perception

Research describing the cellular coding of faces in non-human primates often provides the underlying physiological framework for our understanding of face processing in humans. Models of face perception, explanations of perceptual after-effects from viewing particular types of faces, and interpretation of human neuroimaging data rely on monkey neurophysiological data and the assumption that neurophysiological responses of humans are comparable to those recorded in the non-human primate. Here, we review studies that describe cells that preferentially respond to faces, and assess the link between the physiological characteristics of single cells and social perception. Principally, we describe cells recorded from the non-human primate, although a limited number of cells have been recorded in humans, and are included in order to appraise the validity of non-human physiological data for our understanding of human face and social perception.

Visual adaptation and face perception

The appearance of faces can be strongly affected by the characteristics of faces viewed previously. These perceptual after-effects reflect processes of sensory adaptation that are found throughout the visual system, but which have been considered only relatively recently in the context of higher level perceptual judgements. In this review, we explore the consequences of adaptation for human face perception, and the implications of adaptation for understanding the neural-coding schemes underlying the visual representation of faces. The properties of face after-effects suggest that they, in part, reflect response changes at high and possibly face-specific levels of visual processing. Yet, the form of the after-effects and the norm-based codes that they point to show many parallels with the adaptations and functional organization that are thought to underlie the encoding of perceptual attributes like colour. The nature and basis for human colour vision have been studied extensively, and we draw on ideas and principles that have been developed to account for norms and normalization in colour vision to consider potential similarities and differences in the representation and adaptation of faces.

Identity-specific face adaptation effects: evidence for abstractive face representations

The effects of selective adaptation on familiar face perception were examined. After prolonged exposure to photographs of a celebrity, participants saw a series of ambiguous morphs that were varying mixtures between the face of that person and a different celebrity. Participants judged fewer of the morphs to resemble the celebrity to which they had been adapted, implying that they were now less sensitive to that particular face. Similar results were obtained when the adapting faces were highly dissimilar in viewpoint to the test morphs; when they were presented upside-down; or when they were vertically stretched to three times their normal height. These effects rule out explanations of adaptation effects solely in terms of low-level image-based adaptation. Instead they are consistent with the idea that relatively viewpoint-independent, person-specific adaptation occurred, at the level of either the "Face Recognition Units" or "Person Identity Nodes" in Burton, Bruce and Johnston's (1990) model of face recognition.

Adaptation improves discrimination of face identity

Whether face adaptation confers any advantages to perceptual processing remains an open question. We investigated whether face adaptation can enhance the ability to make fine discriminations in the vicinity of the adapted face. We compared face discrimination thresholds in three adapting conditions: (i) same-face: where adapting and test faces were the same, (ii) different-face: where adapting and test faces differed, and (iii) baseline: where the adapting stimulus was a blank. Discrimination thresholds for morphed identity changes involving the adapted face (same-face) improved compared with those from both the baseline (no-adaptation) and different-face conditions. Since adapting to a face did not alter discrimination performance for other faces, this effect is selective for the facial identity that is adapted. These results indicate a form of gain control to heighten perceptual sensitivity in the vicinity of a currently viewed face, analogous to forms of adaptive gain control at lower levels of the visual system.

Visual Aftereffects for Walking Actions Reveal Underlying Neural Mechanisms for Action Recognition

The results of this study illustrate a new high-level visual aftereffect: Observing actors walking forward, without horizontal translation, makes subsequent actors appear to walk backward, and the opposite effect is obtained after observing backward walking. We used this aftereffect, which cannot be explained by simple low-level adaptation to motion direction, to investigate the properties of neural mechanisms underlying recognition of walking actions. Our results suggest that the perception of walking and the perception of static images of actors in walking postures rely on common brain mechanisms that are primarily object centered, rather than viewer centered, and that are blind to the identity of the actor. These results, obtained with human psychophysical adaptation techniques, support previous evidence accumulated using single-unit recording in nonhuman primates. In addition, these results provide evidence that current models of human action recognition require an object-centered processing stage.

Anti-expression aftereffects reveal prototype-referenced coding of facial expressions

Adaptation is a powerful experimental technique that has recently provided insights into how people encode representations of facial identity. Here, we used this approach to explore the visual representation of facial expressions of emotion. Participants were adapted to anti-expressions of six facial expressions. The participants were then shown an average face and asked to classify the face's expression using one of six basic emotion descriptors. Participants chose the emotion matching the anti-expression they were adapted to significantly more often than they chose any other emotion (e.g., if they were adapted to antifear, they classified the emotion on the average face as fear). The strength of this aftereffect of adaptation decreased as the strength of the anti-expression adapter decreased. These findings provide evidence that visual representations of facial expressions of emotion are coded with reference to a prototype within a multidimensional framework.

Effect of photographic negation on face expression aftereffects

Our visual representation of facial expression is examined in this study: is this representation built from edge information, or does it incorporate surface-based information? To answer this question, photographic negation of grey-scale images is used. Negation preserves edge information whilst disrupting the surface-based information. In two experiments visual aftereffects produced by prolonged viewing of images of facial expressions were measured. This adaptation-based technique allows a behavioural assessment of the characteristics encoded by the neural systems underlying our representation of facial expression. The experiments show that photographic negation of the adapting images results in a profound decrease of expression aftereffect. Our visual representation of facial expression therefore appears to not just be built from edge information, but to also incorporate surface information. The latter allows an appreciation of the 3-D structure of the expressing face that, it is argued, may underpin the subtlety and range of our non-verbal facial communication.

Visual Adaptation to Goal-directed Hand Actions

Prolonged exposure to visual stimuli, or adaptation, often results in an adaptation “aftereffect” which can profoundly distort our perception of subsequent visual stimuli. This technique has been commonly used to investigate mechanisms underlying our perception of simple visual stimuli, and more recently, of static faces. We tested whether humans would adapt to movies of hands grasping and placing different weight objects. After adapting to hands grasping light or heavy objects, subsequently perceived objects appeared relatively heavier, or lighter, respectively. The aftereffects increased logarithmically with adaptation action repetition and decayed logarithmically with time. Adaptation aftereffects also indicated that perception of actions relies predominantly on view-dependent mechanisms. Adapting to one action significantly influenced the perception of the opposite action. These aftereffects can only be explained by adaptation of mechanisms that take into account the presence/absence of the object in the hand. We tested if evidence on action processing mechanisms obtained using visual adaptation techniques confirms underlying neural processing. We recorded monkey superior temporal sulcus (STS) single-cell responses to hand actions. Cells sensitive to grasping or placing typically responded well to the opposite action; cells also responded during different phases of the actions. Cell responses were sensitive to the view of the action and were dependent upon the presence of the object in the scene. We show here that action processing mechanisms established using visual adaptation parallel the neural mechanisms revealed during recording from monkey STS. Visual adaptation techniques can thus be usefully employed to investigate brain mechanisms underlying action perception.

Defining the face processing network: optimization of the functional localizer in fMRI

Functional localizers that contrast brain signal when viewing faces versus objects are commonly used in functional magnetic resonance imaging studies of face processing. However, current protocols do not reliably show all regions of the core system for face processing in all subjects when conservative statistical thresholds are used, which is problematic in the study of single subjects. Furthermore, arbitrary variations in the applied thresholds are associated with inconsistent estimates of the size of face-selective regions-of-interest (ROIs). We hypothesized that the use of more natural dynamic facial images in localizers might increase the likelihood of identifying face-selective ROIs in individual subjects, and we also investigated the use of a method to derive the statistically optimal ROI cluster size independent of thresholds. We found that dynamic facial stimuli were more effective than static stimuli, identifying 98% (versus 72% for static) of ROIs in the core face processing system and 69% (versus 39% for static) of ROIs in the extended face processing system. We then determined for each core face processing ROI, the cluster size associated with maximum statistical face-selectivity, which on average was approximately 50 mm(3) for the fusiform face area, the occipital face area, and the posterior superior temporal sulcus. We suggest that the combination of (a) more robust face-related activity induced by a dynamic face localizer and (b) a cluster-size determination based on maximum face-selectivity increases both the sensitivity and the specificity of the characterization of face-related ROIs in individual subjects.

Effects of alcohol consumption and alcohol expectancy on the categorisation of perceptual cues of emotional expression

RATIONALE

Evidence that alcohol leads to increased aggressive behaviour is equivocal and confounded by evidence that such effects may operate indirectly via expectancy. One mechanism by which alcohol consumption may increase aggressive behaviour is via alterations in the processing of emotional facial cues.

OBJECTIVES

We investigated whether acute alcohol consumption or the expectancy of consuming alcohol (or both) induces differences in the categorisation of ambiguous emotional expressions. We also explored differences between male and female participants, using male and female facial cues of emotional expression.

METHODS

Following consumption of a drink, participants completed a categorisation task in which they had to identify the emotional expression of a facial stimulus. Stimuli were morphed facial images ranging between unambiguously angry and happy expressions (condition 1) or between unambiguously angry and disgusted expressions (condition 2). Participants (N = 96) were randomised to receive an alcoholic or non-alcoholic drink and to be told that they would receive an alcoholic or non-alcoholic drink.

RESULTS

Significant effects of alcohol were obtained in the angry-disgusted task condition, but only when the target facial stimulus was male. Participants tended to categorise male disgusted faces as angry after alcohol, but not after placebo.

CONCLUSIONS

Our data indicate that alcohol consumption may increase the likelihood of an ambiguous but negative facial expression being judged as angry. However, these effects were only observed for male faces and therefore may have been influenced by the greater expectation of aggression in males compared to females. Implications for alcohol-associated aggressive behaviour are discussed.

Effects of acute alcohol consumption on processing of perceptual cues of emotional expression

Alcohol consumption has been associated with increases in aggressive behaviour. However, experimental evidence of a direct association is equivocal, and mechanisms that may underlie this relationship are poorly understood. One mechanism by which alcohol consumption may increase aggressive behaviour is via alterations in processing of emotional facial cues. We investigated the effects of acute alcohol consumption on sensitivity to facial expressions of emotion. Participants attended three experimental sessions where they consumed an alcoholic drink (0.0, 0.2 or 0.4 g/kg), and completed a psychophysical task to distinguish expressive from neutral faces. The level of emotion in the expressive face varied across trials the threshold at which the expressive face was reliably identified and measured. We observed a significant three-way interaction involving emotion, participant sex and alcohol dose. Male participants showed significantly higher perceptual thresholds for sad facial expressions compared with female participants following consumption of the highest dose of alcohol. Our data indicate sex differences in the processing of facial cues of emotional expression following alcohol consumption. There was no evidence that alcohol altered the processing of angry facial expressions. Future studies should examine effects of alcohol expectancy and investigate the effects of alcohol on the miscategorisation of emotional expressions.