Retinotopy of the face aftereffect

Inactivation of face-selective neurons alters eye movements when free viewing faces

During free viewing, faces attract gaze and induce specific fixation patterns corresponding to the facial features. This suggests that neurons encoding the facial features are in the causal chain that steers the eyes. However, there is no physiological evidence to support a mechanistic link between face-encoding neurons in high-level visual areas and the oculomotor system. In this study, we targeted the middle face patches of the inferior temporal (IT) cortex in two macaque monkeys using an functional magnetic resonance imaging (fMRI) localizer. We then utilized muscimol microinjection to unilaterally suppress IT neural activity inside and outside the face patches and recorded eye movements while the animals free viewing natural scenes. Inactivation of the face-selective neurons altered the pattern of eye movements on faces: The monkeys found faces in the scene but neglected the eye contralateral to the inactivation hemisphere. These findings reveal the causal contribution of the high-level visual cortex in eye movements.

Three's a crowd: Fast ensemble perception of first impressions of trustworthiness

Trustworthiness impressions are fundamental social judgements with far-reaching consequences in many aspects of society, including criminal justice, leadership selection and partner preferences. Thus far, most research has focused on facial characteristics that make a face individually appear more or less trustworthy. However, in everyday life, faces are not always perceived in isolation but are often encountered in crowds. It has been proposed that we deal with the large amount of facial information in a group by extracting summary statistics of the crowd, a phenomenon called ensemble perception. Prior research showed that ensemble perception occurs for various facial features, such as emotional expression, facial identity, and attractiveness. Here, we investigated whether observers can integrate the level of trustworthiness from multiple faces to extract an average impression of the crowd. Across four studies, participants were presented with crowds of faces and were asked to report their average level of trustworthiness with an adjustment (Experiment 1) and a rating task (Experiments 2 and 3). Participants were able to extract an ensemble perception of trustworthiness impressions from multiple faces. Moreover, observers were able to form a summary statistic of trustworthiness impressions from a group of faces as quickly as 250 ms (Experiment 4). Taken together, these results demonstrate that ensemble perception can occur at the level of impressions of trustworthiness. Thus, these critical social judgements not only occur for individual faces but are also integrated into a unique ensemble impression of crowds. Our findings contribute to the development of a more ecological approach to the study of trust impressions, since they provide an understanding of trustworthiness judgements not only on an individual level, but on a much broader social group level. Furthermore, our results drive forward new theory because they demonstrate for the first time that ensemble representations cover a broad range of phenomena than previously recognized, including complex high-level facial trait judgements such as trustworthiness impressions.

Inactivation of face selective neurons alters eye movements when free viewing faces

During free viewing, faces attract gaze and induce specific fixation patterns corresponding to the facial features. This suggests that neurons encoding the facial features are in the causal chain that steers the eyes. However, there is no physiological evidence to support a mechanistic link between face encoding neurons in high-level visual areas and the oculomotor system. In this study, we targeted the middle face patches of inferior temporal (IT) cortex in two macaque monkeys using an fMRI localizer. We then utilized muscimol microinjection to unilaterally suppress IT neural activity inside and outside the face patches and recorded eye movements while the animals free viewing natural scenes. Inactivation of the face selective neurons altered the pattern of eye movements on faces: the monkeys found faces in the scene but neglected the eye contralateral to the inactivation hemisphere. These findings reveal the causal contribution of the high-level visual cortex in eye movements.

Significance

It has been shown, for more than half a century, that eye movements follow distinctive patterns when free viewing faces. This suggests causal involvement of the face-encoding visual neurons in the eye movements. However, the literature is scant of evidence for this possibility and has focused mostly on the link between low-level image saliency and eye movements. Here, for the first time, we bring causal evidence showing how face-selective neurons in inferior temporal cortex inform and steer eye movements when free viewing faces.

Preference for horizontal information in faces predicts typical variations in face recognition but is not impaired in developmental prosopagnosia

Face recognition is strongly influenced by the processing of orientation structure in the face image. Faces are much easier to recognize when they are filtered to include only horizontally oriented information compared with vertically oriented information. Here, we investigate whether preferences for horizontal information in faces are related to face recognition abilities in a typical sample (Experiment 1), and whether such preferences are lacking in people with developmental prosopagnosia (DP; Experiment 2). Experiment 1 shows that preferences for horizontal face information are linked to face recognition abilities in a typical sample, with weak evidence of face-selective contributions. Experiment 2 shows that preferences for horizontal face information are comparable in control and DP groups. Our study suggests that preferences for horizontal face information are related to variations in face recognition abilities in the typical range, and that these preferences are not aberrant in DP.

Face identity and facial expression representations with adaptation paradigms: New directions for potential applications

Adaptation and aftereffect are well-known procedures for exploring our neural representation of visual stimuli. It has been reported that they occur in face identity, facial expressions, and low-level visual features. This method has two primary advantages. One is to reveal the common or shared process of faces, that is, the overlapped or discrete representation of face identities or facial expressions. The other is to investigate the coding system or theory of face processing that underlies the ability to recognize faces. This study aims to organize recent research to guide the reader into the field of face adaptation and its aftereffect and to suggest possible future expansions in the use of this paradigm. To achieve this, we reviewed the behavioral short-term aftereffect studies on face identity (i.e., who it is) and facial expressions (i.e., what expressions such as happiness and anger are expressed), and summarized their findings about the neural representation of faces. First, we summarize the basic characteristics of face aftereffects compared to simple visual features to clarify that facial aftereffects occur at a different stage and are not inherited or combinations of low-level visual features. Next, we introduce the norm-based coding hypothesis, which is one of the theories used to represent face identity and facial expressions, and adaptation is a commonly used procedure to examine this. Subsequently, we reviewed studies that applied this paradigm to immature or impaired face recognition (i.e., children and individuals with autism spectrum disorder or prosopagnosia) and examined the relationships between their poor recognition performance and representations. Moreover, we reviewed studies dealing with the representation of non-presented faces and social signals conveyed via faces and discussed that the face adaptation paradigm is also appropriate for these types of examinations. Finally, we summarize the research conducted to date and propose a new direction for the face adaptation paradigm.

The spatial properties of adaptation-induced distance compression

Exposure to a dynamic texture reduces the perceived separation between objects, altering the mapping between physical relations in the environment and their neural representations. Here we investigated the spatial tuning and spatial frame of reference of this aftereffect to understand the stage(s) of processing where adaptation-induced changes occur. In Experiment 1, we measured apparent separation at different positions relative to the adapted area, revealing a strong but tightly tuned compression effect. We next tested the spatial frame of reference of the effect, either by introducing a gaze shift between adaptation and test phase (Experiment 2) or by decoupling the spatial selectivity of adaptation in retinotopic and world-centered coordinates (Experiment 3). Results across the two experiments indicated that both retinotopic and world-centered adaptation effects can occur independently. Spatial attention to the location of the adaptor alone could not account for the world-centered transfer we observed, and retinotopic adaptation did not transfer to world-centered coordinates after a saccade (Experiment 4). Finally, we found that aftereffects in different reference frames have a similar, narrow spatial tuning profile (Experiment 5). Together, our results suggest that the neural representation of local separation resides early in the visual cortex, but it can also be modulated by activity in higher visual areas.

Spatial selectivity in adaptation to gaze direction

A person's focus of attention is conveyed by the direction of their eyes and face, providing a simple visual cue fundamental to social interaction. A growing body of research examines the visual mechanisms that encode the direction of another person's gaze as we observe them. Here we investigate the spatial receptive field properties of these mechanisms, by testing the spatial selectivity of sensory adaptation to gaze direction. Human observers were adapted to faces with averted gaze presented in one visual hemifield, then tested in their perception of gaze direction for faces presented in the same or opposite hemifield. Adaptation caused strong, repulsive perceptual aftereffects, but only for faces presented in the same hemifield as the adapter. This occurred even though adapting and test stimuli were in the same external location across saccades. Hence, there was clear evidence for retinotopic adaptation and a relative lack of either spatiotopic or spatially invariant adaptation. These results indicate that adaptable representations of gaze direction in the human visual system have retinotopic spatial receptive fields. This strategy of coding others' direction of gaze with positional specificity relative to one's own eye position may facilitate key functions of gaze perception, such as socially cued shifts in visual attention.

Examining the Impact of Human Face Stimulus on Shape-Contrast Effects during a Brief Presentation

Shape-contrast effects have been introduced to the investigations into face perception with the aim of exploring face adaptation in the context of norm-based coding. Research has indicated that shape-contrast effects occur even for shapes as complex as the human face. However, whether the complexity of face stimuli alters the magnitude of shape-contrast effects needs to be examined. In this study, emoticons and realistic human faces were used with the original white circle as the test stimuli. The results revealed that the shape-contrast effect was dependent on the stimulus. However, there was no significant difference between the shape-contrast effect evoked by upright faces and that evoked by inverted ones. This suggests that the face stimuli influenced the strength of the shape-contrast effect: the mechanism of this effect involved multiple stages of the visual system related to luminance and complexity, rather than the holistic face perception.

Stimulus uncertainty predicts serial dependence in orientation judgements

How is what you see influenced by what you saw? The visual system may use recent perception to inform responses to current stimuli. This can cause the perception of current stimuli to be attracted toward previous observations, an effect termed serial dependence. This misperception might well be useful in a noisy visual environment, where minor image distortions over time may not actually represent meaningful change. Previous work has suggested that Bayesian perceptual inference may underlie serial dependence. For this to be true, the relative uncertainty associated with both prior and current sensory input should be taken into account. In an experiment manipulating the level of noise present in orientation stimuli, we found an effect of current stimulus uncertainty on serial dependence. We found no good evidence for an effect of previous stimulus uncertainty. Our results provide only partial evidence for the Bayesian interpretation of serial dependence. Non-Bayesian models may provide a better account of the phenomenon.

Temporal stability of stimulus representation increases along rodent visual cortical hierarchies

Cortical representations of brief, static stimuli become more invariant to identity-preserving transformations along the ventral stream. Likewise, increased invariance along the visual hierarchy should imply greater temporal persistence of temporally structured dynamic stimuli, possibly complemented by temporal broadening of neuronal receptive fields. However, such stimuli could engage adaptive and predictive processes, whose impact on neural coding dynamics is unknown. By probing the rat analog of the ventral stream with movies, we uncovered a hierarchy of temporal scales, with deeper areas encoding visual information more persistently. Furthermore, the impact of intrinsic dynamics on the stability of stimulus representations grew gradually along the hierarchy. A database of recordings from mouse showed similar trends, additionally revealing dependencies on the behavioral state. Overall, these findings show that visual representations become progressively more stable along rodent visual processing hierarchies, with an important contribution provided by intrinsic processing.

The human visual system and CNNs can both support robust online translation tolerance following extreme displacements

Visual translation tolerance refers to our capacity to recognize objects over a wide range of different retinal locations. Although translation is perhaps the simplest spatial transform that the visual system needs to cope with, the extent to which the human visual system can identify objects at previously unseen locations is unclear, with some studies reporting near complete invariance over 10 degrees and other reporting zero invariance at 4 degrees of visual angle. Similarly, there is confusion regarding the extent of translation tolerance in computational models of vision, as well as the degree of match between human and model performance. Here, we report a series of eye-tracking studies (total N = 70) demonstrating that novel objects trained at one retinal location can be recognized at high accuracy rates following translations up to 18 degrees. We also show that standard deep convolutional neural networks (DCNNs) support our findings when pretrained to classify another set of stimuli across a range of locations, or when a global average pooling (GAP) layer is added to produce larger receptive fields. Our findings provide a strong constraint for theories of human vision and help explain inconsistent findings previously reported with convolutional neural networks (CNNs).

Parallel Adaptation to Spatially Distinct Distortions

Optical distortions as a visual disturbance are inherent in many optical devices such as spectacles or virtual reality headsets. In such devices, distortions vary spatially across the visual field. In progressive addition lenses, for example, the left and right regions of the lens skew the peripheral parts of the wearers visual field in opposing directions. The human visual system adapts to homogeneous distortions and the respective aftereffects are transferred to non-retinotopic locations. This study investigates simultaneous adaptation to two opposing distortions at different retinotopic locations. Two oppositely skewed natural image sequences were presented to 10 subjects as adaptation stimuli at two distinct locations in the visual field. To do so, subjects were instructed to keep fixation on a target. Eye tracking was used for gaze control. Change of perceived motion direction was measured in a direction identification task. The point of subjective equality (PSE), that is, the angle at which a group of coherently moving dots was perceived as moving horizontal, was determined for both retinal locations. The shift of perceived motion direction was evaluated by comparing PSE before and after adaptation. A significant shift at both retinal locations in the direction of the skew distortion of the corresponding adaptation stimulus is demonstrated. Consequently, parallel adaptation to two opposing distortions in a retinotopic reference frame was confirmed by this study.

Face Adaptation and Face Priming as Tools for Getting Insights Into the Quality of Face Space

During the recognition of faces, the incoming perceptual information is matched against mental representations of familiar faces stored in memory. Face space models describe an abstract concept of face representations and their mental organization, in which facial representations are located on various characteristic dimensions, depending on their specific facial characteristics. However, these models are defined just as incompletely as the general understanding of face recognition. We took two phenomena from face processing to better understand face recognition, and so the nature of face space: face adaptation and face priming. The face literature has mainly focused on face adaptation, largely neglecting face priming when trying to integrate outcomes regarding face recognition into the face space framework. Consequently, the present paper aims to review both phenomena and their contributions to face recognition, representation, and face space.

Are expression aftereffects fully explained by tilt adaptation?

Facial expressions are used as critical social cues in everyday life. Adaptation to expressions causes expression aftereffects. These aftereffects are thought to reflect the operation of face-selective neural mechanisms, and are used by researchers to investigate the nature of those mechanisms. However, recent evidence suggests that expression aftereffects could be at least partially explained by the inheritance of lower-level tilt adaptation through the visual hierarchy. We investigated whether expression aftereffects could be entirely explained by tilt adaptation. Participants completed an expression adaptation task in which we controlled for the influence of tilt by changing the orientation of the adaptor relative to the test stimuli. Although tilt adaptation appeared to make some contribution to the expression aftereffect, robust expression aftereffects still remained after minimizing tilt inheritance, indicating that expression aftereffects cannot be fully explained by tilt adaptation. There was also significant reduction in the expression aftereffects after inverting the adapting face, providing evidence that face-selective processing is involved in these aftereffects.

Brief facial emotion aftereffect occurs earlier for angry than happy adaptation

Prolonged exposure to an emotional face biases our judgement of subsequent face stimulus toward the opposite emotion. This emotion aftereffect has been suggested to occur as early as 35 ms exposure duration in cartoon faces. In the current study, we are interested in investigating the time-course of brief emotional face adaptation, and the relationship between brief emotional face adaptation and prolonged emotional face adaptation. We adapted the subjects from 17 ms to 1000 ms with a happy or angry adapting face. We found that a facial emotion adaptation aftereffect started from 17 ms adapting duration for angry face adaptation, and from 50 ms for happy face adaptation. Factor analysis on the adaptation effects highlighted three different components: brief angry adaptation (17 ms, 34 ms, and 50 ms), prolonged angry adaptation (100 ms and 1000 ms), and happy face adaptation (from 17 ms to 1000 ms). We found that the brief angry face adaptation was negatively associated with the awareness of the adapting face, and the prolonged angry face adaptation was stronger in subjects who perceived the angry adapting face as more negative in valence. Together, these findings suggest that (1) facial emotion adaptation can be induced by brief (17 ms) adapting face presentation; (2) brief angry face adaptation may be related to early visual processing, whereas prolonged angry face adaptation may be related to adaptation at later and higher-level visual emotional processing; and (3) brief and prolonged adaptations may adapt different neural populations. Our findings thus shed light on the current understanding of the neural mechanisms of emotional face adaptation.

Invariance to low-level features and partial transfer over space in the tilt aftereffects evoked by symmetrical patterns

The symmetry axis is the midline that divides a pattern into congruent halves, which is physically nonexistent but evokes tilt aftereffect (TAE). To investigate the cortical correspondence of the symmetry axis, we examined the invariance of symmetry-induced TAE with regard to low-level visual features and the spatial transfer of TAE over visual fields. When the adaptation pattern was rotated and changed sequentially with the orientation of the symmetry axis unchanged, the measured TAE decreased only slightly (18%) compared to stationary patterns. This effect persisted when the adaptation and test patterns were presented in different visual fields. These results indicate that the cortical representation of symmetry is generated independently of low-level features and involves higher-level visual areas.

Adaptation to dynamic faces produces face identity aftereffects

Face aftereffects are well established for static stimuli and have been used extensively as a tool for understanding the neural mechanisms underlying face recognition. It has also been argued that adaptive coding, as demonstrated by face aftereffects, plays a functional role in face recognition by calibrating our face norms to reflect current experience. If aftereffects tap high-level perceptual mechanisms that are critically involved in everyday face recognition then they should also occur for moving faces. Here we asked whether face identity aftereffects can be induced using dynamic adaptors. The face identity aftereffect occurs when adaptation to a particular identity (e.g., Dan) biases subsequent perception toward the opposite identity (e.g., antiDan). We adapted participants to video of real faces that displayed either rigid, non-rigid, or no motion and tested for aftereffects in static antifaces. Adapt and test stimuli differed in size, to minimize low-level adaptation. Aftereffects were found in all conditions, suggesting that face identity aftereffects tap high-level mechanisms important for face recognition. Aftereffects were not significantly reduced in the motion conditions relative to the static condition. Overall, our results support the view that face aftereffects reflect adaptation of high-level mechanisms important for real-world face recognition in which faces are moving.

Sensory and decision-making processes underlying perceptual adaptation

Perceptual systems adapt to their inputs. As a result, prolonged exposure to particular stimuli alters judgments about subsequent stimuli. This phenomenon is commonly assumed to be sensory in origin. Changes in the decision-making process, however, may also be a component of adaptation. Here, we quantify sensory and decision-making contributions to adaptation in a facial expression paradigm. As expected, exposure to happy or sad expressions shifts the psychometric function toward the adaptor. More surprisingly, response times show both an overall decline and an asymmetry, with faster responses opposite the adapting category, implicating a substantial change in the decision-making process. Specifically, we infer that sensory changes from adaptation are accompanied by changes in how much sensory information is accumulated for the two choices. We speculate that adaptation influences implicit expectations about the stimuli one will encounter, causing modifications in the decision-making process as part of a normative response to a change in context.

Rapid Adaptation to the Timbre of Natural Sounds

Timbre, the unique quality of a sound that points to its source, allows us to quickly identify a loved one's voice in a crowd and distinguish a buzzy, bright trumpet from a warm cello. Despite its importance for perceiving the richness of auditory objects, timbre is a relatively poorly understood feature of sounds. Here we demonstrate for the first time that listeners adapt to the timbre of a wide variety of natural sounds. For each of several sound classes, participants were repeatedly exposed to two sounds (e.g., clarinet and oboe, male and female voice) that formed the endpoints of a morphed continuum. Adaptation to timbre resulted in consistent perceptual aftereffects, such that hearing sound A significantly altered perception of a neutral morph between A and B, making it sound more like B. Furthermore, these aftereffects were robust to moderate pitch changes, suggesting that adaptation to timbral features used for object identification drives these effects, analogous to face adaptation in vision.

Adaptation to Skew Distortions of Natural Scenes and Retinal Specificity of Its Aftereffects

Image skew is one of the prominent distortions that exist in optical elements, such as in spectacle lenses. The present study evaluates adaptation to image skew in dynamic natural images. Moreover, the cortical levels involved in skew coding were probed using retinal specificity of skew adaptation aftereffects. Left and right skewed natural image sequences were shown to observers as adapting stimuli. The point of subjective equality (PSE), i.e., the skew amplitude in simple geometrical patterns that is perceived to be unskewed, was used to quantify the aftereffect of each adapting skew direction. The PSE, in a two-alternative forced choice paradigm, shifted toward the adapting skew direction. Moreover, significant adaptation aftereffects were obtained not only at adapted, but also at non-adapted retinal locations during fixation. Skew adaptation information was transferred partially to non-adapted retinal locations. Thus, adaptation to skewed natural scenes induces coordinated plasticity in lower and higher cortical areas of the visual pathway.

An investigation of the spatial selectivity of the duration after-effect

Adaptation to the duration of a visual stimulus causes the perceived duration of a subsequently presented stimulus with a slightly different duration to be skewed away from the adapted duration. This pattern of repulsion following adaptation is similar to that observed for other visual properties, such as orientation, and is considered evidence for the involvement of duration-selective mechanisms in duration encoding. Here, we investigated whether the encoding of duration - by duration-selective mechanisms - occurs early on in the visual processing hierarchy. To this end, we investigated the spatial specificity of the duration after-effect in two experiments. We measured the duration after-effect at adapter-test distances ranging between 0 and 15° of visual angle and for within- and between-hemifield presentations. We replicated the duration after-effect: the test stimulus was perceived to have a longer duration following adaptation to a shorter duration, and a shorter duration following adaptation to a longer duration. Importantly, this duration after-effect occurred at all measured distances, with no evidence for a decrease in the magnitude of the after-effect at larger distances or across hemifields. This shows that adaptation to duration does not result from adaptation occurring early on in the visual processing hierarchy. Instead, it seems likely that duration information is a high-level stimulus property that is encoded later on in the visual processing hierarchy.

Position selectivity in face-sensitive visual cortex to facial and nonfacial stimuli: an fMRI study

BACKGROUND

Evidence for position sensitivity in object-selective visual areas has been building. On one hand, most of the relevant studies have utilized stimuli for which the areas are optimally selective and examine small sections of cortex. On the other hand, visual field maps established with nonspecific stimuli have been found in increasingly large areas of visual cortex, though generally not in areas primarily responsive to faces.

METHODS

fMRI was used to study the position sensitivity of the occipital face area (OFA) and the fusiform face area (FFA) to both standard rotating wedge retinotopic mapping stimuli and quadrant presentations of synthetic facial stimuli. Analysis methods utilized were both typical, that is, mean univariate BOLD signals and multivoxel pattern analysis (MVPA), and novel, that is, distribution of voxels to pattern classifiers and use of responses to nonfacial retinotopic mapping stimuli to classify responses to facial stimuli.

RESULTS

Polar angle sensitivity was exhibited to standard retinotopic mapping stimuli with a stronger contralateral bias in OFA than in FFA, a stronger bias toward the vertical meridian in FFA than in OFA, and a bias across both areas toward the inferior visual field. Contralateral hemispheric lateralization of both areas was again shown using synthetic face stimuli based on univariate BOLD signals, MVPA, and the biased contribution of voxels toward multivariate classifiers discriminating the contralateral visual field. Classifiers based on polar angle responsivity were used to classify the patterns of activation above chance levels to face stimuli in the OFA but not in the FFA.

CONCLUSIONS

Both the OFA and FFA exhibit quadrant sensitivity to face stimuli, though the OFA exhibits greater position responsivity across stimuli than the FFA and includes overlap in the response pattern to the disparate stimulus types. Such biases are consistent with varying position sensitivity along different surfaces of occipito-temporal cortex.

Bidirectional Gender Face Aftereffects: Evidence Against Normative Facial Coding

Facial appearance can be altered, not just by restyling but also by sensory processes. Exposure to a female face can, for instance, make subsequent faces look more masculine than they would otherwise. Two explanations exist. According to one, exposure to a female face renormalizes face perception, making that female and all other faces look more masculine as a consequence—a unidirectional effect. According to that explanation, exposure to a male face would have the opposite unidirectional effect. Another suggestion is that face gender is subject to contrastive aftereffects. These should make some faces look more masculine than the adaptor and other faces more feminine—a bidirectional effect. Here, we show that face gender aftereffects are bidirectional, as predicted by the latter hypothesis. Images of real faces rated as more and less masculine than adaptors at baseline tended to look even more and less masculine than adaptors post adaptation. This suggests that, rather than mental representations of all faces being recalibrated to better reflect the prevailing statistics of the environment, mental operations exaggerate differences between successive faces, and this can impact facial gender perception.

Saccadic remapping of object-selective information

Saccadic remapping, a presaccadic increase in neural activity when a saccade is about to bring an object into a neuron's receptive field, may be crucial for our perception of a stable world. Studies of perception and saccadic remapping, like ours, focus on the presaccadic acquisition of information from the saccade target, with no direct reference to underlying physiology. While information is known to be acquired prior to a saccade, it is unclear whether object-selective or feature-specific information is remapped. To test this, we performed a series of psychophysical experiments in which we presented a peripheral, nonfoveated face as a presaccadic target. The target face disappeared at saccade onset. After making a saccade to the location of the peripheral target face (which was no longer visible), subjects misperceived the expression of a subsequent, foveally presented neutral face as being repelled away from the peripheral presaccadic face target. This effect was similar to a sequential shape contrast or negative aftereffect but required a saccade, because covert attention was not sufficient to generate the illusion. Additional experiments further revealed that inverting the faces disrupted the illusion, suggesting that presaccadic remapping is object-selective and not based on low-level features. Our results demonstrate that saccadic remapping can be an object-selective process, spatially tuned to the target of the saccade and distinct from covert attention in the absence of a saccade.

Transsaccadic processing: stability, integration, and the potential role of remapping

While our frequent saccades allow us to sample the complex visual environment in a highly efficient manner, they also raise certain challenges for interpreting and acting upon visual input. In the present, selective review, we discuss key findings from the domains of cognitive psychology, visual perception, and neuroscience concerning two such challenges: (1) maintaining the phenomenal experience of visual stability despite our rapidly shifting gaze, and (2) integrating visual information across discrete fixations. In the first two sections of the article, we focus primarily on behavioral findings. Next, we examine the possibility that a neural phenomenon known as predictive remapping may provide an explanation for aspects of transsaccadic processing. In this section of the article, we delineate and critically evaluate multiple proposals about the potential role of predictive remapping in light of both theoretical principles and empirical findings.

Optogenetic and pharmacological suppression of spatial clusters of face neurons reveal their causal role in face gender discrimination

Neurons that respond more to images of faces over nonface objects were identified in the inferior temporal (IT) cortex of primates three decades ago. Although it is hypothesized that perceptual discrimination between faces depends on the neural activity of IT subregions enriched with "face neurons," such a causal link has not been directly established. Here, using optogenetic and pharmacological methods, we reversibly suppressed the neural activity in small subregions of IT cortex of macaque monkeys performing a facial gender-discrimination task. Each type of intervention independently demonstrated that suppression of IT subregions enriched in face neurons induced a contralateral deficit in face gender-discrimination behavior. The same neural suppression of other IT subregions produced no detectable change in behavior. These results establish a causal link between the neural activity in IT face neuron subregions and face gender-discrimination behavior. Also, the demonstration that brief neural suppression of specific spatial subregions of IT induces behavioral effects opens the door for applying the technical advantages of optogenetics to a systematic attack on the causal relationship between IT cortex and high-level visual perception.

Object similarity affects the perceptual strategy underlying invariant visual object recognition in rats

In recent years, a number of studies have explored the possible use of rats as models of high-level visual functions. One central question at the root of such an investigation is to understand whether rat object vision relies on the processing of visual shape features or, rather, on lower-order image properties (e.g., overall brightness). In a recent study, we have shown that rats are capable of extracting multiple features of an object that are diagnostic of its identity, at least when those features are, structure-wise, distinct enough to be parsed by the rat visual system. In the present study, we have assessed the impact of object structure on rat perceptual strategy. We trained rats to discriminate between two structurally similar objects, and compared their recognition strategies with those reported in our previous study. We found that, under conditions of lower stimulus discriminability, rat visual discrimination strategy becomes more view-dependent and subject-dependent. Rats were still able to recognize the target objects, in a way that was largely tolerant (i.e., invariant) to object transformation; however, the larger structural and pixel-wise similarity affected the way objects were processed. Compared to the findings of our previous study, the patterns of diagnostic features were: (i) smaller and more scattered; (ii) only partially preserved across object views; and (iii) only partially reproducible across rats. On the other hand, rats were still found to adopt a multi-featural processing strategy and to make use of part of the optimal discriminatory information afforded by the two objects. Our findings suggest that, as in humans, rat invariant recognition can flexibly rely on either view-invariant representations of distinctive object features or view-specific object representations, acquired through learning.

Invariant visual object recognition and shape processing in rats

Invariant visual object recognition is the ability to recognize visual objects despite the vastly different images that each object can project onto the retina during natural vision, depending on its position and size within the visual field, its orientation relative to the viewer, etc. Achieving invariant recognition represents such a formidable computational challenge that is often assumed to be a unique hallmark of primate vision. Historically, this has limited the invasive investigation of its neuronal underpinnings to monkey studies, in spite of the narrow range of experimental approaches that these animal models allow. Meanwhile, rodents have been largely neglected as models of object vision, because of the widespread belief that they are incapable of advanced visual processing. However, the powerful array of experimental tools that have been developed to dissect neuronal circuits in rodents has made these species very attractive to vision scientists too, promoting a new tide of studies that have started to systematically explore visual functions in rats and mice. Rats, in particular, have been the subjects of several behavioral studies, aimed at assessing how advanced object recognition and shape processing is in this species. Here, I review these recent investigations, as well as earlier studies of rat pattern vision, to provide an historical overview and a critical summary of the status of the knowledge about rat object vision. The picture emerging from this survey is very encouraging with regard to the possibility of using rats as complementary models to monkeys in the study of higher-level vision.

Erasing the face after-effect

Perceptual after-effects decay over time at a rate that depends on several factors, such as the duration of adaptation and the duration of the test stimuli. Whether this decay is accelerated by exposure to other faces after adaptation is not known. Our goal was to determine if the appearance of other faces during a delay period after adaptation affected the face identity after-effect. In the first experiment we investigated whether, in the perception of ambiguous stimuli created by morphing between two faces, the repulsive after-effects from adaptation to one face were reduced by brief presentation of the second face in a delay period. We found no effect; however, this may have been confounded by a small attractive after-effect from the interference face. In the second experiment, the interference stimuli were faces unrelated to those used as adaptation stimuli, and we examined after-effects at three different delay periods. This showed a decline in after-effects as the time since adaptation increased, and an enhancement of this decline by the presentation of intervening faces. An exponential model estimated that the intervening faces caused an 85% reduction in the time constant of the after-effect decay. In conclusion, we confirm that face after-effects decline rapidly after adaptation and that exposure to other faces hastens the re-setting of the system.

Retinotopy versus face selectivity in macaque visual cortex

Retinotopic organization is a ubiquitous property of lower-tier visual cortical areas in human and nonhuman primates. In macaque visual cortex, the retinotopic maps extend to higher-order areas in the ventral visual pathway, including area TEO in the inferior temporal (IT) cortex. Distinct regions within IT cortex are also selective to specific object categories such as faces. Here we tested the topographic relationship between retinotopic maps and face-selective patches in macaque visual cortex using high-resolution fMRI and retinotopic face stimuli. Distinct subregions within face-selective patches showed either (1) a coarse retinotopic map of eccentricity and polar angle, (2) a retinotopic bias to a specific location of visual field, or (3) nonretinotopic selectivity. In general, regions along the lateral convexity of IT cortex showed more overlap between retinotopic maps and face selectivity, compared with regions within the STS. Thus, face patches in macaques can be subdivided into smaller patches with distinguishable retinotopic properties.

Intact facial adaptation in autistic adults

Adaptation paradigms seek to bias subsequently viewed stimuli through prolonged exposure to an adapting stimulus, thereby giving rise to an aftereffect. Recent experiments have found that children with autism spectrum disorders (ASD) show reduced facial aftereffects, prompting some researchers to speculate that all individuals with ASD exhibit deficient facial adaptation. However, caution is required when generalizing findings from samples of children with ASD to the wider ASD population. The reduced facial aftereffects seen in child samples may instead reflect delayed or atypical developmental trajectories, whereby individuals with ASD are slower to develop adaptive mechanisms. In the present study, two experiments were conducted to determine whether high-functioning adults with ASD also show diminished aftereffects for facial identity and expression. In Experiment 1, using a procedure that minimized the contribution of low-level retinotopic adaptation, we observed substantial aftereffects comparable to those seen in matched controls, for both facial identity and expression. A similar pattern of results was seen in Experiment 2 using a revised procedure that increased the contribution of retinotopic adaptation to the facial aftereffects observed. That adults with autism can show robust facial aftereffects raises the possibility that group differences are seen only at particular points during development, and may not be a lifelong feature of the condition.

Seeing a haptically explored face: visual facial-expression aftereffect from haptic adaptation to a face

Current views on face perception assume that the visual system receives only visual facial signals. However, I show that the visual perception of faces is systematically biased by adaptation to a haptically explored face. Recently, face aftereffects (FAEs; the altered perception of faces after adaptation to a face) have been demonstrated not only in visual perception but also in haptic perception; therefore, I combined the two FAEs to examine whether the visual system receives face-related signals from the haptic modality. I found that adaptation to a haptically explored facial expression on a face mask produced a visual FAE for facial expression. This cross-modal FAE was not due to explicitly imaging a face, response bias, or adaptation to local features. Furthermore, FAEs transferred from vision to haptics. These results indicate that visual face processing depends on substrates adapted by haptic faces, which suggests that face processing relies on shared representation underlying cross-modal interactions.

Shape similarity, better than semantic membership, accounts for the structure of visual object representations in a population of monkey inferotemporal neurons

The anterior inferotemporal cortex (IT) is the highest stage along the hierarchy of visual areas that, in primates, processes visual objects. Although several lines of evidence suggest that IT primarily represents visual shape information, some recent studies have argued that neuronal ensembles in IT code the semantic membership of visual objects (i.e., represent conceptual classes such as animate and inanimate objects). In this study, we investigated to what extent semantic, rather than purely visual information, is represented in IT by performing a multivariate analysis of IT responses to a set of visual objects. By relying on a variety of machine-learning approaches (including a cutting-edge clustering algorithm that has been recently developed in the domain of statistical physics), we found that, in most instances, IT representation of visual objects is accounted for by their similarity at the level of shape or, more surprisingly, low-level visual properties. Only in a few cases we observed IT representations of semantic classes that were not explainable by the visual similarity of their members. Overall, these findings reassert the primary function of IT as a conveyor of explicit visual shape information, and reveal that low-level visual properties are represented in IT to a greater extent than previously appreciated. In addition, our work demonstrates how combining a variety of state-of-the-art multivariate approaches, and carefully estimating the contribution of shape similarity to the representation of object categories, can substantially advance our understanding of neuronal coding of visual objects in cortex.

On the hierarchical inheritance of aftereffects in the visual system

The emotion perceived in a face can be influenced by prior exposure to a face expressing a different emotion. Here we show that displacement along a particular emotional axis, that encoding happiness and sadness, can be effected solely by a systematic change in the angle, at the center of the mouth, between the left and right halves of the mouth. We then demonstrate that adaptation to a face with the mouth distorted to change this angle, such that the face expresses an emotion on this axis, causes a face with a neutral expression to be perceived as having the opposite expression. By abstracting the mouths from the faces and examining the magnitude of the angle aftereffects in the mouths alone and in an unfamiliar orientation, we show that the magnitudes of the angle aftereffects are sufficient to account for the changes in perceived emotion in the faces. Further, by applying the distortion to the mouths asymmetrically so that the distortion is manifested by a change in orientation of the mouth stimulus rather than a change in angle, we show that the magnitude of the aftereffect can be predicted by the local tilt aftereffect. We argue, therefore, that the aftereffects of emotion are due to misperception of morphology of the face and that the misperception is due to the local change in perceived orientation due to the systematic application of the tilt aftereffect in a tilt aftereffect field. All adaptation experiments were performed using stimuli that were either high-pass or low-pass filtered for spatial frequency. Results showed that the spatial frequency specificity of the aftereffects was the same for the face, angled mouth, and oriented mouth stimuli, lending further support to the hypothesis that the aftereffects are instantiated in processes early in the visual cortex and that the aftereffects assumed to be higher level are, in fact, inherited.

Redundancy effects in the processing of emotional faces

How does the visual system represent the ensemble statistics of visual objects? This question has received intense interest in vision research, yet most studies have focused on the extraction of mean statistics rather than its dispersion. This study focuses on another aspect of ensemble statistics: the redundancy of the sample. In two experiments, participants were faster judging the facial expression and gender of multiple faces than a single face. The redundancy gain was equivalent for multiple identical faces and for multiple faces of different identities. To test whether the redundancy gain was due to increased strength in perceptual representation, we measured the magnitude of facial expression aftereffects. The aftereffects were equivalent when induced by a single face and by four identical faces, ruling out increased perceptual strength as an explanation for the redundancy gain. We conclude that redundant faces facilitate perception by enhancing the robustness of representation of each face.

Face aftereffects predict individual differences in face recognition ability

Face aftereffects are widely studied on the assumption that they provide a useful tool for investigating face-space coding of identity. However, a long-standing issue concerns the extent to which face aftereffects originate in face-level processes as opposed to earlier stages of visual processing. For example, some recent studies failed to find atypical face aftereffects in individuals with clinically poor face recognition. We show that in individuals within the normal range of face recognition abilities, there is an association between face memory ability and a figural face aftereffect that is argued to reflect the steepness of broadband-opponent neural response functions in underlying face-space. We further show that this correlation arises from face-level processing, by reporting results of tests of nonface memory and nonface aftereffects. We conclude that face aftereffects can tap high-level face-space, and that face-space coding differs in quality between individuals and contributes to face recognition ability.

Position sensitivity in the visual word form area

Seeing words involves the activity of neural circuitry within a small region in human ventral temporal cortex known as the visual word form area (VWFA). It is widely asserted that VWFA responses, which are essential for skilled reading, do not depend on the visual field position of the writing (position invariant). Such position invariance supports the hypothesis that the VWFA analyzes word forms at an abstract level, far removed from specific stimulus features. Using functional MRI pattern-classification techniques, we show that position information is encoded in the spatial pattern of VWFA responses. A right-hemisphere homolog (rVWFA) shows similarly position-sensitive responses. Furthermore, electrophysiological recordings in the human brain show position-sensitive VWFA response latencies. These findings show that position-sensitive information is present in the neural circuitry that conveys visual word form information to language areas. The presence of position sensitivity in the VWFA has implications for how word forms might be learned and stored within the reading circuitry.

Neural correlates of after-effects caused by adaptation to multiple face displays

Adaptation to a given face leads to face-related, specific after-effects. Recently, this topic has attracted a lot of attention because it clearly shows that adaptation occurs even at the higher stages of visual cortical processing. However, during our every-day life, faces do not appear in isolation, rather they are usually surrounded by other stimuli. Here, we used psychophysical and fMRI adaptation methods to test whether humans adapt to the gender properties of a composite multiple face stimulus as well. As adaptors we used stimuli composed of eight different individual faces, positioned peripherally in a ring around a fixation mark. We found that the gender discrimination of a subsequent centrally presented target face is significantly biased as a result of long-term adaptation to either male or female multiple face stimuli. Similar to our previous results with single-face adaptors (Kovács et al. in Neuroimage 43(1):156-164, 2008), a concurrent functional magnetic resonance imaging adaptation experiment revealed the strongest blood oxygen level-dependent signal adaptation bilaterally in the fusiform face area. Our results suggest that humans extract the statistical features of the multiple face stimulus and this process occurs at the level of occipito-temporal face processing.

Transformation-tolerant object recognition in rats revealed by visual priming

Successful use of rodents as models for studying object vision crucially depends on the ability of their visual system to construct representations of visual objects that tolerate (i.e., remain relatively unchanged with respect to) the tremendous changes in object appearance produced, for instance, by size and viewpoint variation. Whether this is the case is still controversial, despite some recent demonstration of transformation-tolerant object recognition in rats. In fact, it remains unknown to what extent such a tolerant recognition has a spontaneous, perceptual basis, or, alternatively, mainly reflects learning of arbitrary associative relations among trained object appearances. In this study, we addressed this question by training rats to categorize a continuum of morph objects resulting from blending two object prototypes. The resulting psychometric curve (reporting the proportion of responses to one prototype along the morph line) served as a reference when, in a second phase of the experiment, either prototype was briefly presented as a prime, immediately before a test morph object. The resulting shift of the psychometric curve showed that recognition became biased toward the identity of the prime. Critically, this bias was observed also when the primes were transformed along a variety of dimensions (i.e., size, position, viewpoint, and their combination) that the animals had never experienced before. These results indicate that rats spontaneously perceive different views/appearances of an object as similar (i.e., as instances of the same object) and argue for the existence of neuronal substrates underlying formation of transformation-tolerant object representations in rats.

How does the brain solve visual object recognition?

Mounting evidence suggests that 'core object recognition,' the ability to rapidly recognize objects despite substantial appearance variation, is solved in the brain via a cascade of reflexive, largely feedforward computations that culminate in a powerful neuronal representation in the inferior temporal cortex. However, the algorithm that produces this solution remains poorly understood. Here we review evidence ranging from individual neurons and neuronal populations to behavior and computational models. We propose that understanding this algorithm will require using neuronal and psychophysical data to sift through many computational models, each based on building blocks of small, canonical subnetworks with a common functional goal.

Selectivity of face distortion aftereffects for differences in expression or gender

The perceived configuration of a face can be strongly biased by prior adaptation to a face with a distorted configuration. These aftereffects have been found to be weaker when the adapt and test faces differ along a number of dimensions. We asked whether the adaptation shows more transfer between faces that share a common identity, by comparing the strength of aftereffects when the adapt and test faces differed either in expression (a configural change in the same face identity) or gender (a configural change between identities). Observers adapted to expanded or contracted images of either male or female faces with either happy or fearful expressions, and then judged the perceived configuration in either the same faces or faces with a different gender and/or expression. The adaptation included exposure to a single face (e.g., expanded happy) or to alternated faces where the distortion was contingent on the attribute (e.g., expanded happy versus contracted fearful). In all cases the aftereffects showed strong transfer and thus only weak selectivity. However, selectivity was equal or stronger for the change in expression than gender. Our results thus suggest that the distortion aftereffects between faces can be weakly modulated by both variant and invariant attributes of the face.

Selectivity of face aftereffects for expressions and anti-expressions

Adapting to a facial expression can alter the perceived expression of subsequently viewed faces. However, it remains unclear whether this adaptation affects each expression independently or transfers from one expression to another, and whether this transfer impedes or enhances responses to a different expression. To test for these interactions, we probed the basic expressions of anger, fear, happiness, sadness, surprise, and disgust, adapting to one expression and then testing on all six. Each expression was varied in strength by morphing it with a common neutral facial expression. Observers determined the threshold level required to correctly identify each expression, before or after adapting to a face with a neutral or intense expression. The adaptation was strongly selective for the adapting category; responses to the adapting expression were reduced, while other categories showed little consistent evidence of either suppression or facilitation. In a second experiment we instead compared adaptation to each expression and its anti-expression. The latter are defined by the physically complementary facial configuration, yet appear much more ambiguous as expressions. In this case, for most expressions the opposing faces produced aftereffects of opposite sign in the perceived expression. These biases suggest that the adaptation acts in part by shifting the perceived neutral point for the facial configuration. This is consistent with the pattern of renormalization suggested for adaptation to other facial attributes, and thus may reflect a generic level of configural coding. However, for most categories aftereffects were stronger for expressions than anti-expressions, pointing to the possible influence of an additional component of the adaptation at sites that explicitly represent facial expressions. At either level our results are consistent with other recent work in suggesting that the six expressions are defined by dimensions that are largely independently normalized by adaptation, possibly because the facial configurations conveying different expressions vary in independent ways.

Adaptive face space coding in congenital prosopagnosia: typical figural aftereffects but abnormal identity aftereffects

People with congenital prosopagnosia (CP) report difficulty recognising faces in everyday life and perform poorly on face recognition tests. Here, we investigate whether impaired adaptive face space coding might contribute to poor face recognition in CP. To pinpoint how adaptation may affect face processing, a group of CPs and matched controls completed two complementary face adaptation tasks: the figural aftereffect, which reflects adaptation to general distortions of shape, and the identity aftereffect, which directly taps the mechanisms involved in the discrimination of different face identities. CPs displayed a typical figural aftereffect, consistent with evidence that they are able to process some shape-based information from faces, e.g., cues to discriminate sex. CPs also demonstrated a significant identity aftereffect. However, unlike controls, CPs impression of the identity of the neutral average face was not significantly shifted by adaptation, suggesting that adaptive coding of identity is abnormal in CP. In sum, CPs show reduced aftereffects but only when the task directly taps the use of face norms used to code individual identity. This finding of a reduced face identity aftereffect in individuals with severe face recognition problems is consistent with suggestions that adaptive coding may have a functional role in face recognition.