Rapid and dynamic processing of face pareidolia in the human brain

The human brain is specialized for face processing, yet we sometimes perceive illusory faces in objects. It is unknown whether these natural errors of face detection originate from a rapid process based on visual features or from a slower, cognitive re-interpretation. Here we use a multifaceted approach to understand both the spatial distribution and temporal dynamics of illusory face representation in the brain by combining functional magnetic resonance imaging and magnetoencephalography neuroimaging data with model-based analysis. We find that the representation of illusory faces is confined to occipital-temporal face-selective visual cortex. The temporal dynamics reveal a striking evolution in how illusory faces are represented relative to human faces and matched objects. Illusory faces are initially represented more similarly to real faces than matched objects are, but within ~250 ms, the representation transforms, and they become equivalent to ordinary objects. This is consistent with the initial recruitment of a broadly-tuned face detection mechanism which privileges sensitivity over selectivity.

Pervasive influence of idiosyncratic associative biases during facial emotion recognition

Facial morphology has been shown to influence perceptual judgments of emotion in a way that is shared across human observers. Here we demonstrate that these shared associations between facial morphology and emotion coexist with strong variations unique to each human observer. Interestingly, a large part of these idiosyncratic associations does not vary on short time scales, emerging from stable inter-individual differences in the way facial morphological features influence emotion recognition. Computational modelling of decision-making and neural recordings of electrical brain activity revealed that both shared and idiosyncratic face-emotion associations operate through a common biasing mechanism rather than an increased sensitivity to face-associated emotions. Together, these findings emphasize the underestimated influence of idiosyncrasies on core social judgments and identify their neuro-computational signatures.

Perception and Processing of Faces in the Human Brain Is Tuned to Typical Feature Locations

Faces are salient social stimuli whose features attract a stereotypical pattern of fixations. The implications of this gaze behavior for perception and brain activity are largely unknown. Here, we characterize and quantify a retinotopic bias implied by typical gaze behavior toward faces, which leads to eyes and mouth appearing most often in the upper and lower visual field, respectively. We found that the adult human visual system is tuned to these contingencies. In two recognition experiments, recognition performance for isolated face parts was better when they were presented at typical, rather than reversed, visual field locations. The recognition cost of reversed locations was equal to ∼60% of that for whole face inversion in the same sample. Similarly, an fMRI experiment showed that patterns of activity evoked by eye and mouth stimuli in the right inferior occipital gyrus could be separated with significantly higher accuracy when these features were presented at typical, rather than reversed, visual field locations. Our findings demonstrate that human face perception is determined not only by the local position of features within a face context, but by whether features appear at the typical retinotopic location given normal gaze behavior. Such location sensitivity may reflect fine-tuning of category-specific visual processing to retinal input statistics. Our findings further suggest that retinotopic heterogeneity might play a role for face inversion effects and for the understanding of conditions affecting gaze behavior toward faces, such as autism spectrum disorders and congenital prosopagnosia.

SIGNIFICANCE STATEMENT

Faces attract our attention and trigger stereotypical patterns of visual fixations, concentrating on inner features, like eyes and mouth. Here we show that the visual system represents face features better when they are shown at retinal positions where they typically fall during natural vision. When facial features were shown at typical (rather than reversed) visual field locations, they were discriminated better by humans and could be decoded with higher accuracy from brain activity patterns in the right occipital face area. This suggests that brain representations of face features do not cover the visual field uniformly. It may help us understand the well-known face-inversion effect and conditions affecting gaze behavior toward faces, such as prosopagnosia and autism spectrum disorders.

The Deceptively Simple N170 Reflects Network Information Processing Mechanisms Involving Visual Feature Coding and Transfer Across Hemispheres

A key to understanding visual cognition is to determine “where”, “when”, and “how” brain responses reflect the processing of the specific visual features that modulate categorization behavior—the “what”. The N170 is the earliest Event-Related Potential (ERP) that preferentially responds to faces. Here, we demonstrate that a paradigmatic shift is necessary to interpret the N170 as the product of an information processing network that dynamically codes and transfers face features across hemispheres, rather than as a local stimulus-driven event. Reverse-correlation methods coupled with information-theoretic analyses revealed that visibility of the eyes influences face detection behavior. The N170 initially reflects coding of the behaviorally relevant eye contralateral to the sensor, followed by a causal communication of the other eye from the other hemisphere. These findings demonstrate that the deceptively simple N170 ERP hides a complex network information processing mechanism involving initial coding and subsequent cross-hemispheric transfer of visual features.

Stimulus features coded by single neurons of a macaque body category selective patch

Body category-selective regions of the primate temporal cortex respond to images of bodies, but it is unclear which fragments of such images drive single neurons' responses in these regions. Here we applied the Bubbles technique to the responses of single macaque middle superior temporal sulcus (midSTS) body patch neurons to reveal the image fragments the neurons respond to. We found that local image fragments such as extremities (limbs), curved boundaries, and parts of the torso drove the large majority of neurons. Bubbles revealed the whole body in only a few neurons. Neurons coded the features in a manner that was tolerant to translation and scale changes. Most image fragments were excitatory but for a few neurons both inhibitory and excitatory fragments (opponent coding) were present in the same image. The fragments we reveal here in the body patch with Bubbles differ from those suggested in previous studies of face-selective neurons in face patches. Together, our data indicate that the majority of body patch neurons respond to local image fragments that occur frequently, but not exclusively, in bodies, with a coding that is tolerant to translation and scale. Overall, the data suggest that the body category selectivity of the midSTS body patch depends more on the feature statistics of bodies (e.g., extensions occur more frequently in bodies) than on semantics (bodies as an abstract category).

Dissociation of part-based and integrated neural responses to faces by means of electroencephalographic frequency tagging

In order to isolate the repetition suppression effects for each part of a whole-face stimulus, the left and right halves of face stimuli were flickered at different frequency rates (5.88 or 7.14 Hz), changing or not changing identity at every stimulation cycle. The human electrophysiological (electroencephalographic) responses to each face half increased in amplitude when different rather than repeated face half identities were presented at every stimulation cycle. Contrary to the repetition suppression effects for whole faces, which are usually found over the right occipito-temporal cortex, these part-based repetition suppression effects were found on all posterior electrode sites and were unchanged when the two face halves were manipulated by separation, lateral misalignment, or inversion. In contrast, intermodulation components (e.g. 7.14-5.88 = 1.26 Hz) were found mainly over the right occipito-temporal cortex and were significantly reduced following the aforementioned manipulations. In addition, the intermodulation components decreased substantially for face halves belonging to different identities, which form a less coherent face than when they belong to the same face identity. These observations provide objective evidence for dissociation between part-based and integrated (i.e. holistic/configural) responses to faces in the human brain, suggesting that only responses to integrated face parts reflect high-level, possibly face-specific, representations.

Phase noise reveals early category-specific modulation of the event-related potentials

Previous studies have found that the amplitude of the early event-related potential (ERP) components evoked by faces, such as N170 and P2, changes systematically as a function of noise added to the stimuli. This change has been linked to an increased perceptual processing demand and to enhanced difficulty in perceptual decision making about faces. However, to date it has not yet been tested whether noise manipulation affects the neural correlates of decisions about face and non-face stimuli similarly. To this end, we measured the ERPs for faces and cars at three different phase noise levels. Subjects performed the same two-alternative age-discrimination task on stimuli chosen from young–old morphing continua that were created from faces as well as cars and were calibrated to lead to similar performances at each noise-level. Adding phase noise to the stimuli reduced performance and enhanced response latency for the two categories to the same extent. Parallel to that, phase noise reduced the amplitude and prolonged the latency of the face-specific N170 component. The amplitude of the P1 showed category-specific noise dependence: it was enhanced over the right hemisphere for cars and over the left hemisphere for faces as a result of adding phase noise to the stimuli, but remained stable across noise levels for cars over the left and for faces over the right hemisphere. Moreover, noise modulation altered the category-selectivity of the N170, while the P2 ERP component, typically associated with task decision difficulty, was larger for the more noisy stimuli regardless of stimulus category. Our results suggest that the category-specificity of noise-induced modulations of ERP responses starts at around 100 ms post-stimulus.

Event-related potentials to unattended changes in facial expressions: detection of regularity violations or encoding of emotions?

Visual mismatch negativity (vMMN), a component in event-related potentials (ERPs), can be elicited when rarely presented “deviant” facial expressions violate regularity formed by repeated “standard” faces. vMMN is observed as differential ERPs elicited between the deviant and standard faces. It is not clear, however, whether differential ERPs to rare emotional faces interspersed with repeated neutral ones reflect true vMMN (i.e., detection of regularity violation) or merely encoding of the emotional content in the faces. Furthermore, a face-sensitive N170 response, which reflects structural encoding of facial features, can be modulated by emotional expressions. Owing to its similar latency and scalp topography with vMMN, these two components are difficult to separate. We recorded ERPs to neutral, fearful, and happy faces in two different stimulus presentation conditions in adult humans. For the oddball condition group, frequently presented neutral expressions (p = 0.8) were rarely replaced by happy or fearful expressions (p = 0.1), whereas for the equiprobable condition group, fearful, happy, and neutral expressions were presented with equal probability (p = 0.33). Independent component analysis (ICA) revealed two prominent components in both stimulus conditions in the relevant latency range and scalp location. A component peaking at 130 ms post stimulus showed a difference in scalp topography between the oddball (bilateral) and the equiprobable (right-dominant) conditions. The other component, peaking at 170 ms post stimulus, showed no difference between the conditions. The bilateral component at the 130-ms latency in the oddball condition conforms to vMMN. Moreover, it was distinct from N170 which was modulated by the emotional expression only. The present results suggest that future studies on vMMN to facial expressions should take into account possible confounding effects caused by the differential processing of the emotional expressions as such.

Single-trial ERP analysis reveals facial expression category in a three-stage scheme

Emotional faces are salient stimuli that play a critical role in social interactions. Following up on previous research suggesting that the event-related potentials (ERPs) show differential amplitudes in response to various facial expressions, the current study used trial-to-trial variability assembled from six discriminating ERP components to predict the facial expression categories in individual trials. In an experiment involved 17 participants, fearful trials were differentiated from non-fearful trials as early as the intervals of N1 and P1, with a mean predictive accuracy of 87%. Single-trial features in the occurrence of N170 and vertex positive potential could distinguish between emotional and neutral expressions (accuracy=90%). Finally, the trials associated with fearful, happy, and neutral faces were completely separated during the window of N3 and P3 (accuracy=83%). These categorization findings elucidated the temporal evolution of facial expression extraction, and demonstrated that the spatio-temporal characteristics of single-trial ERPs can distinguish facial expressions according to a three-stage scheme, with "fear popup," "emotional/unemotional discrimination," and "complete separation" as processing stages. This work constitutes the first examination of neural processing dynamics beyond multitrial ERP averaging, and directly relates the prediction performance of single-trial classifiers to the progressive brain functions of emotional face discrimination.

Decoding face categories in diagnostic subregions of primary visual cortex

Higher visual areas in the occipitotemporal cortex contain discrete regions for face processing, but it remains unclear if V1 is modulated by top-down influences during face discrimination, and if this is widespread throughout V1 or localized to retinotopic regions processing task-relevant facial features. Employing functional magnetic resonance imaging (fMRI), we mapped the cortical representation of two feature locations that modulate higher visual areas during categorical judgements – the eyes and mouth. Subjects were presented with happy and fearful faces, and we measured the fMRI signal of V1 regions processing the eyes and mouth whilst subjects engaged in gender and expression categorization tasks. In a univariate analysis, we used a region-of-interest-based general linear model approach to reveal changes in activation within these regions as a function of task. We then trained a linear pattern classifier to classify facial expression or gender on the basis of V1 data from ‘eye’ and ‘mouth’ regions, and from the remaining non-diagnostic V1 region. Using multivariate techniques, we show that V1 activity discriminates face categories both in local ‘diagnostic’ and widespread ‘non-diagnostic’ cortical subregions. This indicates that V1 might receive the processed outcome of complex facial feature analysis from other cortical (i.e. fusiform face area, occipital face area) or subcortical areas (amygdala).

Precedence of the eye region in neural processing of faces

Functional magnetic resonance imaging (fMRI) has revealed multiple subregions in monkey inferior temporal cortex (IT) that are selective for images of faces over other objects. The earliest of these subregions, the posterior lateral face patch (PL), has not been studied previously at the neurophysiological level. Perhaps not surprisingly, we found that PL contains a high concentration of "face-selective" cells when tested with standard image sets comparable to those used previously to define the region at the level of fMRI. However, we here report that several different image sets and analytical approaches converge to show that nearly all face-selective PL cells are driven by the presence of a single eye in the context of a face outline. Most strikingly, images containing only an eye, even when incorrectly positioned in an outline, drove neurons nearly as well as full-face images, and face images lacking only this feature led to longer latency responses. Thus, bottom-up face processing is relatively local and linearly integrates features-consistent with parts-based models-grounding investigation of how the presence of a face is first inferred in the IT face processing hierarchy.

Spatially pooled contrast responses predict neural and perceptual similarity of naturalistic image categories

The visual world is complex and continuously changing. Yet, our brain transforms patterns of light falling on our retina into a coherent percept within a few hundred milliseconds. Possibly, low-level neural responses already carry substantial information to facilitate rapid characterization of the visual input. Here, we computationally estimated low-level contrast responses to computer-generated naturalistic images, and tested whether spatial pooling of these responses could predict image similarity at the neural and behavioral level. Using EEG, we show that statistics derived from pooled responses explain a large amount of variance between single-image evoked potentials (ERPs) in individual subjects. Dissimilarity analysis on multi-electrode ERPs demonstrated that large differences between images in pooled response statistics are predictive of more dissimilar patterns of evoked activity, whereas images with little difference in statistics give rise to highly similar evoked activity patterns. In a separate behavioral experiment, images with large differences in statistics were judged as different categories, whereas images with little differences were confused. These findings suggest that statistics derived from low-level contrast responses can be extracted in early visual processing and can be relevant for rapid judgment of visual similarity. We compared our results with two other, well- known contrast statistics: Fourier power spectra and higher-order properties of contrast distributions (skewness and kurtosis). Interestingly, whereas these statistics allow for accurate image categorization, they do not predict ERP response patterns or behavioral categorization confusions. These converging computational, neural and behavioral results suggest that statistics of pooled contrast responses contain information that corresponds with perceived visual similarity in a rapid, low-level categorization task.

Humans excel in rapid and accurate processing of visual scenes. However, it is unclear which computations allow the visual system to convert light hitting the retina into a coherent representation of visual input in a rapid and efficient way. Here we used simple, computer-generated image categories with similar low-level structure as natural scenes to test whether a model of early integration of low-level information can predict perceived category similarity. Specifically, we show that summarized (spatially pooled) responses of model neurons covering the entire visual field (the population response) to low-level properties of visual input (contrasts) can already be informative about differences in early visual evoked activity as well as behavioral confusions of these categories. These results suggest that low-level population responses can carry relevant information to estimate similarity of controlled images, and put forward the exciting hypothesis that the visual system may exploit these responses to rapidly process real natural scenes. We propose that the spatial pooling that allows for the extraction of this information may be a plausible first step in extracting scene gist to form a rapid impression of the visual input.

Systematic, balancing gradients in neuron density and number across the primate isocortex

The cellular and areal organization of the cerebral cortex impacts how it processes and integrates information. How that organization emerges and how best to characterize it has been debated for over a century. Here we demonstrate and describe in the isocortices of seven primate species a pronounced anterior-to-posterior gradient in the density of neurons and in the number of neurons under a unit area of the cortical surface. Our findings assert that the cellular architecture of the primate isocortex is neither arranged uniformly nor into discrete patches with an arbitrary spatial arrangement. Rather, it exhibits striking systematic variation. We conjecture that these gradients, which establish the basic landscape that richer areal and cellular structure is built upon, result from developmental patterns of cortical neurogenesis which are conserved across species. Moreover, we propose a functional consequence: that the gradient in neurons per unit of cortical area fosters the integration and dimensional reduction of information along its ascent through sensory areas and toward frontal cortex.

Internal representations for face detection: an application of noise-based image classification to BOLD responses

What basic visual structures underlie human face detection and how can we extract such structures directly from the amplitude of neural responses elicited by face processing? Here, we address these issues by investigating an extension of noise-based image classification to BOLD responses recorded in high-level visual areas. First, we assess the applicability of this classification method to such data and, second, we explore its results in connection with the neural processing of faces. To this end, we construct luminance templates from white noise fields based on the response of face-selective areas in the human ventral cortex. Using behaviorally and neurally-derived classification images, our results reveal a family of simple but robust image structures subserving face representation and detection. Thus, we confirm the role played by classical face selective regions in face detection and we help clarify the representational basis of this perceptual function. From a theory standpoint, our findings support the idea of simple but highly diagnostic neurally-coded features for face detection. At the same time, from a methodological perspective, our work demonstrates the ability of noise-based image classification in conjunction with fMRI to help uncover the structure of high-level perceptual representations.

Brain mapping biomarkers of socio-emotional processing in schizophrenia

The Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) initiative has formed with the expressed intent of identifying constructs and paradigms that would identify biomarkers of psychosis. The manipulation of these biomarkers would serve as targets for treatment interventions. The second phase of CNTRICS consisted of critical discussions evaluating brain mapping (functional neuroimaging and brain electrical activity) paradigms as biomarkers to measure specific constructs. Among the constructs identified in, CNTRICS I was socio-emotional processing, specifically focused on affect recognition. Here, we provide a critical appraisal of the ability of candidate socio-emotional tasks to identify putative biomarkers and recommendations for future directions in this rapidly moving research domain.

Faces in the mist: illusory face and letter detection

We report three behavioral experiments on the spatial characteristics evoking illusory face and letter detection. False detections made to pure noise images were analyzed using a modified reverse correlation method in which hundreds of observers rated a modest number of noise images (480) during a single session. This method was originally developed for brain imaging research, and has been used in a number of fMRI publications, but this is the first report of the behavioral classification images. In Experiment 1 illusory face detection occurred in response to scattered dark patches throughout the images, with a bias to the left visual field. This occurred despite the use of a fixation cross and expectations that faces would be centered. In contrast, illusory letter detection (Experiment 2) occurred in response to centrally positioned dark patches. Experiment 3 included an oval in all displays to spatially constrain illusory face detection. With the addition of this oval the classification image revealed an eyes/nose/mouth pattern. These results suggest that face detection is triggered by a minimal face-like pattern even when these features are not centered in visual focus.

Quantifying the Time Course of Visual Object Processing Using ERPs: It's Time to Up the Game

Hundreds of studies have investigated the early ERPs to faces and objects using scalp and intracranial recordings. The vast majority of these studies have used uncontrolled stimuli, inappropriate designs, peak measurements, poor figures, and poor inferential and descriptive group statistics. These problems, together with a tendency to discuss any effect p < 0.05 rather than to report effect sizes, have led to a research field very much qualitative in nature, despite its quantitative inspirations, and in which predictions do not go beyond condition A > condition B. Here we describe the main limitations of face and object ERP research and suggest alternative strategies to move forward. The problems plague intracranial and surface ERP studies, but also studies using more advanced techniques – e.g., source space analyses and measurements of network dynamics, as well as many behavioral, fMRI, TMS, and LFP studies. In essence, it is time to stop amassing binary results and start using single-trial analyses to build models of visual perception.

Stereotype-based modulation of person perception

A core social-psychological question is how cultural stereotypes shape our encounters with other people. While there is considerable evidence to suggest that unexpected targets-such as female airline pilots and male nurses-impact the inferential and memorial aspects of person construal, it has yet to be established if early perceptual operations are similarly sensitive to the stereotype-related status of individuals. To explore this issue, the current investigation measured neural activity while participants made social (i.e., sex categorization) and non-social (i.e., dot detection) judgments about men and women portrayed in expected and unexpected occupations. When participants categorized the stimuli according to sex, stereotype-inconsistent targets elicited increased activity in cortical areas associated with person perception and conflict resolution. Comparable effects did not emerge during a non-social judgment task. These findings begin to elucidate how and when stereotypic beliefs modulate the formation of person percepts in the brain.

Asymmetrical use of eye information from faces following unilateral amygdala damage

The human amygdalae are involved in processing visual information about the eyes within faces, and play an essential role in the use of information from the eye region of the face in order to judge emotional expressions, as well as in directing gaze to the eyes in conversations with real people. However, the roles played here by the left and right amygdala individually remain unknown. Here we investigated this question by applying the 'Bubbles' method, which asks viewers to discriminate facial emotions from randomly sampled small regions of a face, to 23 neurological participants with focal, unilateral amygdala damage (10 to the right amygdala). We found a statistically significant asymmetry in the use of eye information when comparing those with unilateral left lesions to those with unilateral right lesions, specifically during emotion judgments. The findings have implications for the amygdala's role in emotion recognition and gaze direction during face processing.

Decoding task-based attentional modulation during face categorization

Attention is a neurocognitive mechanism that selects task-relevant sensory or mnemonic information to achieve current behavioral goals. Attentional modulation of cortical activity has been observed when attention is directed to specific locations, features, or objects. However, little is known about how high-level categorization task set modulates perceptual representations. In the current study, observers categorized faces by gender (male vs. female) or race (Asian vs. White). Each face was perceptually ambiguous in both dimensions, such that categorization of one dimension demanded selective attention to task-relevant information within the face. We used multivoxel pattern classification to show that task-specific modulations evoke reliably distinct spatial patterns of activity within three face-selective cortical regions (right fusiform face area and bilateral occipital face areas). This result suggests that patterns of activity in these regions reflect not only stimulus-specific (i.e., faces vs. houses) responses but also task-specific (i.e., race vs. gender) attentional modulation. Furthermore, exploratory whole-brain multivoxel pattern classification (using a searchlight procedure) revealed a network of dorsal fronto-parietal regions (left middle frontal gyrus and left inferior and superior parietal lobule) that also exhibit distinct patterns for the two task sets, suggesting that these regions may represent abstract goals during high-level categorization tasks.

Dynamics of trimming the content of face representations for categorization in the brain

To understand visual cognition, it is imperative to determine when, how and with what information the human brain categorizes the visual input. Visual categorization consistently involves at least an early and a late stage: the occipito-temporal N170 event related potential related to stimulus encoding and the parietal P300 involved in perceptual decisions. Here we sought to understand how the brain globally transforms its representations of face categories from their early encoding to the later decision stage over the 400 ms time window encompassing the N170 and P300 brain events. We applied classification image techniques to the behavioral and electroencephalographic data of three observers who categorized seven facial expressions of emotion and report two main findings: (1) over the 400 ms time course, processing of facial features initially spreads bilaterally across the left and right occipito-temporal regions to dynamically converge onto the centro-parietal region; (2) concurrently, information processing gradually shifts from encoding common face features across all spatial scales (e.g., the eyes) to representing only the finer scales of the diagnostic features that are richer in useful information for behavior (e.g., the wide opened eyes in ‘fear’; the detailed mouth in ‘happy’). Our findings suggest that the brain refines its diagnostic representations of visual categories over the first 400 ms of processing by trimming a thorough encoding of features over the N170, to leave only the detailed information important for perceptual decisions over the P300.

How the brain uses visual information to construct representations of categories is a central question of cognitive neuroscience. With our methods we visualize how the brain transforms its representations of facial expressions. Using electroencephalographic data, we analyze how representations change over the first 450 ms of processing both in feature content (e.g., which aspects of the face, such as the eyes or the mouth are represented across time) and level of detail. We show that facial expressions are initially encoded with most of their features (i.e., mouth and eyes) across all levels of details in the occipito-temporal regions. In a later phase, we show that a gradual reorganization of representations occurs, whereby only task relevant face features are kept (e.g., the mouth in “happy”) at only the finest level of details. We describe this elimination of irrelevant and redundant information as ‘trimming’. We suggest that this may be an example of the brain optimizing categorical representations.