Not just the norm: exemplar-based models also predict face aftereffects

Investigating the neural effects of typicality and predictability for face and object stimuli

The brain calibrates itself based on the past stimulus diet, which makes frequently observed stimuli appear as typical (as opposed to uncommon stimuli, which appear as distinctive). Based on predictive processing theory, the brain should be more "prepared" for typical exemplars, because these contain information that has been encountered frequently, allowing it to economically represent items of that category. Thus, one could ask whether predictability and typicality of visual stimuli interact, or rather act in an additive manner. We adapted the design by Egner and colleagues (2010), who used cues to induce expectations about stimulus category (face vs. chair) occurrence during an orthogonal inversion detection task. We measured BOLD responses with fMRI in 35 participants. First, distinctive stimuli always elicited stronger responses than typical ones in all ROIs, and our whole-brain directional contrasts for the effects of typicality and distinctiveness converge with previous findings. Second and importantly, we could not replicate the interaction between category and predictability reported by Egner et al. (2010), which casts doubt on whether cueing designs are ideal to elicit reliable predictability effects. Third, likely as a consequence of the lack of predictability effects, we found no interaction between predictability and typicality in any of the four tested regions (bilateral fusiform face areas, lateral occipital complexes) when considering both categories, nor in the whole brain. We discuss the issue of replicability in neuroscience and sketch an agenda for how future studies might address the same question.

Are we really Bayesian? Probabilistic inference shows sub-optimal knowledge transfer

Numerous studies have found that the Bayesian framework, which formulates the optimal integration of the knowledge of the world (i.e. prior) and current sensory evidence (i.e. likelihood), captures human behaviours sufficiently well. However, there are debates regarding whether humans use precise but cognitively demanding Bayesian computations for behaviours. Across two studies, we trained participants to estimate hidden locations of a target drawn from priors with different levels of uncertainty. In each trial, scattered dots provided noisy likelihood information about the target location. Participants showed that they learned the priors and combined prior and likelihood information to infer target locations in a Bayes fashion. We then introduced a transfer condition presenting a trained prior and a likelihood that has never been put together during training. How well participants integrate this novel likelihood with their learned prior is an indicator of whether participants perform Bayesian computations. In one study, participants experienced the newly introduced likelihood, which was paired with a different prior, during training. Participants changed likelihood weighting following expected directions although the degrees of change were significantly lower than Bayes-optimal predictions. In another group, the novel likelihoods were never used during training. We found people integrated a new likelihood within (interpolation) better than the one outside (extrapolation) the range of their previous learning experience and they were quantitatively Bayes-suboptimal in both. We replicated the findings of both studies in a validation dataset. Our results showed that Bayesian behaviours may not always be achieved by a full Bayesian computation. Future studies can apply our approach to different tasks to enhance the understanding of decision-making mechanisms.

Biases in the spectral amplitude distribution of a natural scene modulate horizontal size perception

Introduction

Visual perception is a complex process that involves the analysis of different spatial and temporal features of the visual environment. One critical aspect of this process is adaptation, which allows the visual system to adjust its sensitivity to specific features based on the context of the environment. Numerous theories highlight the significance of the visual scene and its spectral properties in perceptual and adaptation mechanisms. For example, size perception is known to be influenced by the spatial frequency content of the visual scene. Nonetheless, several inquiries still exist, including how specific spectral properties of the scene play a role in size perception and adaptation mechanisms.

Methods

In this study, we explore aftereffects on size perception following adaptation to a natural scene with a biased spectral amplitude distribution. Twenty participants had to manually estimate the horizontal size of a projected rectangle after adaptation to three visually biased conditions: vertical-biased, non-biased, and horizontal-biased. Size adaptation aftereffects were quantified by comparing the perceptual responses from the non-biased condition with the vertical- and horizontal-biased conditions.

Results

We found size perception shifts which were contingent upon the specific orientation and spatial frequency distribution inherent in the amplitude spectra of the adaptation stimuli. Particularly, adaptation to vertical-biased produced a horizontal enlargement, while adaptation to horizontal-biased generated a decrease in the horizontal size perception of the rectangle. On average, size perception was modulated by 5-6%.

Discussion

These findings provide supporting evidence for the hypothesis that the neural mechanisms responsible for processing spatial frequency channels are involved in the encoding and perception of size information. The implications for neural mechanisms underlying spatial frequency and size information encoding are discussed.

Differences between high and low performers in face recognition in electrophysiological correlates of face familiarity and distance-to-norm

Valentine's influential norm-based multidimensional face-space model (nMDFS) predicts that perceived distinctiveness of a face increases with its distance to the norm. Occipito-temporal event-related potentials (ERPs) have been recently shown to respond selectively to variations in distance-to-norm (P200) or familiarity (N250, late negativity), respectively (Wuttke & Schweinberger, 2019). Despite growing evidence on interindividual differences in face perception skills at the behavioral level, little research has focused on their electrophysiological correlates. To reveal potential interindividual differences in face spaces, we contrasted high and low performers in face recognition in regards to distance-to-norm (P200) and familiarity (N250). We replicated both the P200 distance-to-norm and the N250 familiarity effect. Importantly, we observed: i) reduced responses in low compared to high performers of face recognition, especially in terms of smaller distance-to-norm effects in the P200, possibly indicating less 'expanded' face spaces in low compared to high performers; ii) increased N250 responses to familiar original faces in high performers, suggesting more robust face identity representations. In summary, these findings suggest the contribution of both early norm-based face coding and robust face representations to individual face recognition skills, and indicate that ERPs can offer a promising route to understand individual differences in face perception and their neurocognitive correlates.

Facial typicality and attractiveness reflect an ideal dimension of face structure

Face perception and recognition are important processes for social interaction and communication among humans, so understanding how faces are mentally represented and processed has major implications. At the same time, faces are just some of the many stimuli that we encounter in our everyday lives. Therefore, more general theories of how we represent objects might also apply to faces. Contemporary research on the mental representation of faces has centered on two competing theoretical frameworks that arose from more general categorization research: prototype-based face representation and exemplar-based face representation. Empirically distinguishing between these frameworks is difficult and neither one has been ruled out. In this paper, we advance this area of research in three ways. First, we introduce two additional frameworks for mental representation of categories, varying abstraction and ideal representation, which have not been applied to face perception and recognition before. Second, we fit formal computational models of all four of these theories to human perceptual judgments of the typicality and attractiveness (a strong correlate of typicality) of 100 young adult Caucasian female faces, with the models expressed within a face space derived from facial similarity judgments via multidimensional scaling. Third, we predict the perceived typicality and attractiveness of the faces using these models and compare the predictive performance of each to the empirical data. We found that of all four models, the ideal representation model provided the best account of perceived typicality and attractiveness for the present set of faces, although all models showed discrepancies from the empirical data. These findings demonstrate the relevance of mental categorization processes for representing faces.

Face dissimilarity judgments are predicted by representational distance in morphable and image-computable models

Discerning the subtle differences between individuals’ faces is crucial for social functioning. It requires us not only to solve general challenges of object recognition (e.g., invariant recognition over changes in view or lighting) but also to be attuned to the specific ways in which face structure varies. Three-dimensional morphable models based on principal component analyses of real faces provide descriptions of statistical differences between faces, as well as tools to generate novel faces. We rendered large sets of realistic face pairs from such a model and collected similarity and same/different identity judgments. The statistical model predicted human perception as well as state-of-the-art image-computable neural networks. Results underscore the statistical tuning of face encoding.

Human vision is attuned to the subtle differences between individual faces. Yet we lack a quantitative way of predicting how similar two face images look and whether they appear to show the same person. Principal component–based three-dimensional (3D) morphable models are widely used to generate stimuli in face perception research. These models capture the distribution of real human faces in terms of dimensions of physical shape and texture. How well does a “face space” based on these dimensions capture the similarity relationships humans perceive among faces? To answer this, we designed a behavioral task to collect dissimilarity and same/different identity judgments for 232 pairs of realistic faces. Stimuli sampled geometric relationships in a face space derived from principal components of 3D shape and texture (Basel face model [BFM]). We then compared a wide range of models in their ability to predict the data, including the BFM from which faces were generated, an active appearance model derived from face photographs, and image-computable models of visual perception. Euclidean distance in the BFM explained both dissimilarity and identity judgments surprisingly well. In a comparison against 16 diverse models, BFM distance was competitive with representational distances in state-of-the-art deep neural networks (DNNs), including novel DNNs trained on BFM synthetic identities or BFM latents. Models capturing the distribution of face shape and texture across individuals are not only useful tools for stimulus generation. They also capture important information about how faces are perceived, suggesting that human face representations are tuned to the statistical distribution of faces.

Caricatured facial movements enhance perception of emotional facial expressions

Although faces “in the wild” constantly undergo complicated movements, humans adeptly perceive facial identity and expression. Previous studies, focusing mainly on identity, used photographic caricature to show that distinctive form increases perceived dissimilarity. We tested whether distinctive facial movements showed similar effects, and we focussed on both perception of expression and identity. We caricatured the movements of an animated computer head, using physical motion metrics extracted from videos. We verified that these “ground truth” metrics showed the expected effects: Caricature increased physical dissimilarity between faces differing in expression and those differing in identity. Like the ground truth dissimilarity, participants’ dissimilarity perception was increased by caricature when faces differed in expression. We found these perceived dissimilarities to reflect the “representational geometry” of the ground truth. However, neither of these findings held for faces differing in identity. These findings replicated across two paradigms: pairwise ratings and multiarrangement. In a final study, motion caricature did not improve recognition memory for identity, whether manipulated at study or test. We report several forms of converging evidence for spatiotemporal caricature effects on dissimilarity perception of different expressions. However, more work needs to be done to discover what identity-specific movements can enhance face identification.

From likely to likable: The role of statistical typicality in human social assessment of faces

Humans readily form social impressions, such as attractiveness and trustworthiness, from a stranger's facial features. Understanding the provenance of these impressions has clear scientific importance and societal implications. Motivated by the efficient coding hypothesis of brain representation, as well as Claude Shannon's theoretical result that maximally efficient representational systems assign shorter codes to statistically more typical data (quantified as log likelihood), we suggest that social "liking" of faces increases with statistical typicality. Combining human behavioral data and computational modeling, we show that perceived attractiveness, trustworthiness, dominance, and valence of a face image linearly increase with its statistical typicality (log likelihood). We also show that statistical typicality can at least partially explain the role of symmetry in attractiveness perception. Additionally, by assuming that the brain focuses on a task-relevant subset of facial features and assessing log likelihood of a face using those features, our model can explain the "ugliness-in-averageness" effect found in social psychology, whereby otherwise attractive, intercategory faces diminish in attractiveness during a categorization task.

Adaptation aftereffects reveal how categorization training changes the encoding of face identity

Previous research suggests that learning to categorize faces along a novel dimension changes the perceptual representation of such dimension, increasing its discriminability, its invariance, and the information used to identify faces varying along the dimension. A common interpretation of these results is that categorization training promotes the creation of novel dimensions, rather than simply the enhancement of already existing representations. Here, we trained a group of participants to categorize faces that varied along two morphing dimensions, one of them relevant to the categorization task and the other irrelevant to the task. An untrained group did not receive such categorization training. In three experiments, we used face adaptation aftereffects to explore how categorization training changes the encoding of face identities at the extremes of the category-relevant dimension and whether such training produces encoding of the category-relevant dimension as a preferred direction in face space. The pattern of results suggests that categorization training enhances the already existing norm-based coding of face identity, rather than creating novel category-relevant representations. We formalized this conclusion in a model that explains the most important results in our experiments and serves as a working hypothesis for future work in this area.

The Face of Image Reconstruction: Progress, Pitfalls, Prospects

Recent research has demonstrated that neural and behavioral data acquired in response to viewing face images can be used to reconstruct the images themselves. However, the theoretical implications, promises, and challenges of this direction of research remain unclear. We evaluate the potential of this research for elucidating the visual representations underlying face recognition. Specifically, we outline complementary and converging accounts of the visual content, the representational structure, and the neural dynamics of face processing. We illustrate how this research addresses fundamental questions in the study of normal and impaired face recognition, and how image reconstruction provides a powerful framework for uncovering face representations, for unifying multiple types of empirical data, and for facilitating both theoretical and methodological progress.

Face Recognition: Canonical Mechanisms at Multiple Timescales

Adaptation is ubiquitous in the nervous system, and many possible computational roles have been discussed. A new functional imaging study suggests that, in face recognition, the learning of 'norm faces' and adaptation resulting in perceptual after-effects depend on the same mechanism.

Memory and Perception-based Facial Image Reconstruction

Visual memory for faces has been extensively researched, especially regarding the main factors that influence face memorability. However, what we remember exactly about a face, namely, the pictorial content of visual memory, remains largely unclear. The current work aims to elucidate this issue by reconstructing face images from both perceptual and memory-based behavioural data. Specifically, our work builds upon and further validates the hypothesis that visual memory and perception share a common representational basis underlying facial identity recognition. To this end, we derived facial features directly from perceptual data and then used such features for image reconstruction separately from perception and memory data. Successful levels of reconstruction were achieved in both cases for newly-learned faces as well as for familiar faces retrieved from long-term memory. Theoretically, this work provides insights into the content of memory-based representations while, practically, it may open the path to novel applications, such as computer-based 'sketch artists'.

Intensity dependence in high-level facial expression adaptation aftereffect

Perception of a facial expression can be altered or biased by a prolonged viewing of other facial expressions, known as the facial expression adaptation aftereffect (FEAA). Recent studies using antiexpressions have demonstrated a monotonic relation between the magnitude of the FEAA and adaptor extremity, suggesting that facial expressions are opponent coded and represented continuously from one expression to its antiexpression. However, it is unclear whether the opponent-coding scheme can account for the FEAA between two facial expressions. In the current study, we demonstrated that the magnitude of the FEAA between two facial expressions increased monotonically as a function of the intensity of adapting facial expressions, consistent with the predictions based on the opponent-coding model. Further, the monotonic increase in the FEAA occurred even when the intensity of an adapting face was too weak for its expression to be recognized. These results together suggest that multiple facial expressions are encoded and represented by balanced activity of neural populations tuned to different facial expressions.

Adaptive Coding and Face Recognition

Face adaptation generates striking face aftereffects, but is this adaptation useful? The answer appears to be yes, with several lines of evidence suggesting that it contributes to our face-recognition ability. Adaptation to face identity is reduced in a variety of clinical populations with impaired face recognition. In addition, individual differences in face adaptation are linked to face-recognition ability in typical adults. People who adapt more readily to new faces are better at recognizing faces. This link between adaptation and recognition holds for both identity and expression recognition. Adaptation updates face norms, which represent the typical or average properties of the faces we experience. By using these norms to code how faces differ from average, the visual system can make explicit the distinctive information that we need to recognize faces. Thus, adaptive norm-based coding may help us to discriminate and recognize faces despite their similarity as visual patterns.

Preference for facial averageness: Evidence for a common mechanism in human and macaque infants

Human adults and infants show a preference for average faces, which could stem from a general processing mechanism and may be shared among primates. However, little is known about preference for facial averageness in monkeys. We used a comparative developmental approach and eye-tracking methodology to assess visual attention in human and macaque infants to faces naturally varying in their distance from a prototypical face. In Experiment 1, we examined the preference for faces relatively close to or far from the prototype in 12-month-old human infants with human adult female faces. Infants preferred faces closer to the average than faces farther from it. In Experiment 2, we measured the looking time of 3-month-old rhesus macaques (Macaca mulatta) viewing macaque faces varying in their distance from the prototype. Like human infants, macaque infants looked longer to faces closer to the average. In Experiments 3 and 4, both species were presented with unfamiliar categories of faces (i.e., macaque infants tested with adult macaque faces; human infants and adults tested with infant macaque faces) and showed no prototype preferences, suggesting that the prototypicality effect is experience-dependent. Overall, the findings suggest a common processing mechanism across species, leading to averageness preferences in primates.

Modelling individual difference in visual categorization

Recent years has seen growing interest in understanding, characterizing, and explaining individual differences in visual cognition. We focus here on individual differences in visual categorization. Categorization is the fundamental visual ability to group different objects together as the same kind of thing. Research on visual categorization and category learning has been significantly informed by computational modeling, so our review will focus both on how formal models of visual categorization have captured individual differences and how individual difference have informed the development of formal models. We first examine the potential sources of individual differences in leading models of visual categorization, providing a brief review of a range of different models. We then describe several examples of how computational models have captured individual differences in visual categorization. This review also provides a bit of an historical perspective, starting with models that predicted no individual differences, to those that captured group differences, to those that predict true individual differences, and to more recent hierarchical approaches that can simultaneously capture both group and individual differences in visual categorization. Via this selective review, we see how considerations of individual differences can lead to important theoretical insights into how people visually categorize objects in the world around them. We also consider new directions for work examining individual differences in visual categorization.

Face-Space: A Unifying Concept in Face Recognition Research

The concept of a multidimensional psychological space, in which faces can be represented according to their perceived properties, is fundamental to the modern theorist in face processing. Yet the idea was not clearly expressed until 1991. The background that led to the development of face-space is explained, and its continuing influence on theories of face processing is discussed. Research that has explored the properties of the face-space and sought to understand caricature, including facial adaptation paradigms, is reviewed. Face-space as a theoretical framework for understanding the effect of ethnicity and the development of face recognition is evaluated. Finally, two applications of face-space in the forensic setting are discussed. From initially being presented as a model to explain distinctiveness, inversion, and the effect of ethnicity, face-space has become a central pillar in many aspects of face processing. It is currently being developed to help us understand adaptation effects with faces. While being in principle a simple concept, face-space has shaped, and continues to shape, our understanding of face perception.

The Importance of Formalizing Computational Models of Face Adaptation Aftereffects

Face adaptation is widely used as a means to probe the neural representations that support face recognition. While the theories that relate face adaptation to behavioral aftereffects may seem conceptually simple, our work has shown that testing computational instantiations of these theories can lead to unexpected results. Instantiating a model of face adaptation not only requires specifying how faces are represented and how adaptation shapes those representations but also specifying how decisions are made, translating hidden representational states into observed responses. Considering the high-dimensionality of face representations, the parallel activation of multiple representations, and the non-linearity of activation functions and decision mechanisms, intuitions alone are unlikely to succeed. If the goal is to understand mechanism, not simply to examine the boundaries of a behavioral phenomenon or correlate behavior with brain activity, then formal computational modeling must be a component of theory testing. To illustrate, we highlight our recent computational modeling of face adaptation aftereffects and discuss how models can be used to understand the mechanisms by which faces are recognized.

The perception of a face can be greater than the sum of its parts

Holistic processing is often used as a construct to characterize face recognition. An important recent study by Gold, Mundy, and Tjan (2012) quantified holistic processing by computing a facial-feature integration index derived from an ideal observer model. This index was mathematically defined as the ratio of the psychophysical contrast sensitivities squared for recognizing a whole face versus the sum of contrast sensitivities squared for individual face parts (left eye, right eye, nose, and mouth). They observed that this index was not significantly different from 1, leading to the provocative conclusion that the perception of a face is no more than the sum of its parts. What may not be obvious to all readers of this work is that these conclusions were based on a collection of faces that shared essentially the same configuration of face parts. We tested whether the facial-feature integration index would also equal 1 when faces have a range of configurations mirroring the range of variability in real-world faces, using the same experimental procedure and calculating the same integration index as Gold et al. When tested on faces with the same configuration, we also observed an integration index similar to what Gold et al. reported. But when tested on faces with variable configurations, we observed an integration index significantly greater than 1. Combing our results with those of Gold et al. further clarifies the theoretical construct of holistic processing in face recognition and what it means for the whole to be greater than the sum of its parts.

Visual Adaptation

Sensory systems continuously mold themselves to the widely varying contexts in which they must operate. Studies of these adaptations have played a long and central role in vision science. In part this is because the specific adaptations remain a powerful tool for dissecting vision, by exposing the mechanisms that are adapting. That is, "if it adapts, it's there." Many insights about vision have come from using adaptation in this way, as a method. A second important trend has been the realization that the processes of adaptation are themselves essential to how vision works, and thus are likely to operate at all levels. That is, "if it's there, it adapts." This has focused interest on the mechanisms of adaptation as the target rather than the probe. Together both approaches have led to an emerging insight of adaptation as a fundamental and ubiquitous coding strategy impacting all aspects of how we see.

Touching on face space: comparing visual and haptic processing of face shapes

The idea that faces are represented within a structured face space (Valentine Quarterly Journal of Experimental Psychology 43: 161-204, 1991) has gained considerable experimental support, from both physiological and perceptual studies. Recent work has also shown that faces can even be recognized haptically-that is, from touch alone. Although some evidence favors congruent processing strategies in the visual and haptic processing of faces, the question of how similar the two modalities are in terms of face processing remains open. Here, this question was addressed by asking whether there is evidence for a haptic face space, and if so, how it compares to visual face space. For this, a physical face space was created, consisting of six laser-scanned individual faces, their morphed average, 50%-morphs between two individual faces, as well as 50%-morphs of the individual faces with the average, resulting in a set of 19 faces. Participants then rated either the visual or haptic pairwise similarity of the tangible 3-D face shapes. Multidimensional scaling analyses showed that both modalities extracted perceptual spaces that conformed to critical predictions of the face space framework, hence providing support for similar processing of complex face shapes in haptics and vision. Despite the overall similarities, however, systematic differences also emerged between the visual and haptic data. These differences are discussed in the context of face processing and complex-shape processing in vision and haptics.

Adding years to your life (or at least looking like it): a simple normalization underlies adaptation to facial age

Adaptation has been widely used to probe how experience shapes the visual encoding of faces, but the pattern of perceptual changes produced by adaptation and the neural mechanisms these imply remain poorly characterized. We explored how adaptation alters the perceived age of faces, a fundamental facial attribute which can uniquely and reliably be scaled by observers. This allowed us to measure how adaptation to one age level affected the full continuum of perceived ages. Participants guessed the ages of faces ranging from 18-89, before or after adapting to a different set of faces composed of younger, older, or middle-aged adults. Adapting to young or old faces induced opposite linear shifts in perceived age that were independent of the model's age. Specifically, after adapting to younger or older faces, faces of all ages appeared 2 to 3 years older or younger, respectively. In contrast, middle-aged adaptors induced no aftereffects. This pattern suggests that adaptation leads to a simple and uniform renormalization of age perception, and is consistent with a norm-based neural code for the mechanisms mediating the perception of facial age.

Category-specific face prototypes are emerging, but not yet mature, in 5-year-old children

Adults' expertise in face recognition has been attributed to norm-based coding. Moreover, adults possess separable norms for a variety of face categories (e.g., race, sex, age) that appear to enhance recognition by reducing redundancy in the information shared by faces and ensuring that only relevant dimensions are used to encode faces from a given category. Although 5-year-old children process own-race faces using norm-based coding, little is known about the organization and refinement of their face space. The current study investigated whether 5-year-olds rely on category-specific norms and whether experience facilitates the development of dissociable face prototypes. In Experiment 1, we examined whether Chinese 5-year-olds show race-contingent opposing aftereffects and the extent to which aftereffects transfer across face race among Caucasian and Chinese 5-year-olds. Both participant races showed partial transfer of aftereffects across face race; however, there was no evidence for race-contingent opposing aftereffects. To examine whether experience facilitates the development of category-specific prototypes, we investigated whether race-contingent aftereffects are present among Caucasian 5-year-olds with abundant exposure to Chinese faces (Experiment 2) and then tested separate groups of 5-year-olds with two other categories with which they have considerable experience: sex (male/female faces) and age (adult/child faces) (Experiment 3). Across all three categories, 5-year-olds showed no category-contingent opposing aftereffects. These results demonstrate that 5 years of age is a stage characterized by minimal separation in the norms and associated coding dimensions used for faces from different categories and suggest that refinement of the mechanisms that underlie expert face processing occurs throughout childhood.

Toward a unified model of face and object recognition in the human visual system

Our understanding of the mechanisms and neural substrates underlying visual recognition has made considerable progress over the past 30 years. During this period, accumulating evidence has led many scientists to conclude that objects and faces are recognised in fundamentally distinct ways, and in fundamentally distinct cortical areas. In the psychological literature, in particular, this dissociation has led to a palpable disconnect between theories of how we process and represent the two classes of object. This paper follows a trend in part of the recognition literature to try to reconcile what we know about these two forms of recognition by considering the effects of learning. Taking a widely accepted, self-organizing model of object recognition, this paper explains how such a system is affected by repeated exposure to specific stimulus classes. In so doing, it explains how many aspects of recognition generally regarded as unusual to faces (holistic processing, configural processing, sensitivity to inversion, the other-race effect, the prototype effect, etc.) are emergent properties of category-specific learning within such a system. Overall, the paper describes how a single model of recognition learning can and does produce the seemingly very different types of representation associated with faces and objects.