Electrophysiological studies of human face perception. II: Response properties of face-specific potentials generated in occipitotemporal cortex

Temporal Order Judgements of Dynamic Gaze Stimuli Reveal a Postdictive Prioritisation of Averted Over Direct Shifts

We studied temporal order judgements (TOJs) of gaze shift behaviours and evaluated the impact of gaze direction (direct and averted gaze) and face context information (both eyes set within a single face or each eye within two adjacent hemifaces) on TOJ performance measures. Avatar faces initially gazed leftwards or rightwards (Starting Gaze Direction). This was followed by sequential and independent left and right eye gaze shifts with various amounts of stimulus onset asynchrony. Gaze shifts could be either Matching (both eyes end up pointing direct or averted) or Mismatching (one eye ends up pointing direct, the other averted). Matching shifts revealed an attentional cueing mechanism, where TOJs were biased in favour of the eye lying in the hemispace cued by the avatar’s Starting Gaze Direction. For example, the left eye was more likely to be judged as shifting first when the avatar initially gazed toward the left side of the screen. Mismatching shifts showed biased TOJs in favour of the eye performing the averted shift, but only in the context of two separate hemifaces that does not violate expectations of directional gaze shift congruency. This suggests a postdictive inferential strategy that prioritises eye movements based on the type of gaze shift, independently of where attention is initially allocated. Averted shifts are prioritised over direct, as these might signal the presence of behaviourally relevant information in the environment.

Tuning face perception with electrical stimulation of the fusiform gyrus

The fusiform gyrus (FG) is an important node in the face processing network, but knowledge of its causal role in face perception is currently limited. Recent work demonstrated that high frequency stimulation applied to the FG distorts the perception of faces in human subjects (Parvizi et al. []: J Neurosci 32:14915-14920). However, the timing of this process in the FG relative to stimulus onset and the spatial extent of FG's role in face perception are unknown. Here, we investigate the causal role of the FG in face perception by applying precise, event-related electrical stimulation (ES) to higher order visual areas including the FG in six human subjects undergoing intracranial monitoring for epilepsy. We compared the effects of single brief (100 μs) electrical pulses to the FG and non-face-selective visual areas on the speed and accuracy of detecting distorted faces. Brief ES applied to face-selective sites did not affect accuracy but significantly increased the reaction time (RT) of detecting face distortions. Importantly, RT was altered only when ES was applied 100ms after visual onset and in face-selective but not place-selective sites. Furthermore, ES applied to face-selective areas decreased the amplitude of visual evoked potentials and high gamma power over this time window. Together, these results suggest that ES of face-selective regions within a critical time window induces a delay in face perception. These findings support a temporally and spatially specific causal role of face-selective areas and signify an important link between electrophysiology and behavior in face perception. Hum Brain Mapp 38:2830-2842, 2017. © 2017 Wiley Periodicals, Inc.

Holistic versus feature-based binding in the medial temporal lobe

A central question for cognitive neuroscience is how feature-combinations that give rise to episodic/source memories are encoded in the brain. Although there is much evidence that the hippocampus (HIP) is involved in feature binding, and some evidence that other brain regions are as well, there is relatively little evidence about the nature of the resulting representations in different brain regions. We used multivoxel pattern analysis (MVPA) to investigate how feature combinations might be represented, contrasting two possibilities, feature-based versus holistic. Participants viewed stimuli that were composed of three source features - a person (face or body), a scene (indoor or outdoor), and an object (bike or luggage) - which were combined to make eight unique stimulus identities. We reasoned that regions that can classify the eight identities (a multiclass classification) but not the individual features (a binary classification) likely have a holistic representation of each identity. In contrast, regions that can classify the eight identities and can classify each feature are likely to contain feature-based representations of these identities. To further probe the extent of feature-based or holistic classification in each region, we developed and validated a novel approach that directly compares binary and multiclass classification. We found clear evidence for holistic representation in the parahippocampal cortex (PHC), consistent with theories that posit that pattern-separation-like binding mechanisms are not unique to the HIP. Further clarifying the mechanisms of feature binding should benefit from systematic comparisons of multi-feature representations and whether they vary with task, type of stimulus, and/or experience.

A Possible Role of Prolonged Whirling Episodes on Structural Plasticity of the Cortical Networks and Altered Vertigo Perception: The Cortex of Sufi Whirling Dervishes

Although minutes of a spinning episode may induce vertigo in the healthy human, as a result of a possible perceptional plasticity, Sufi Whirling Dervishes (SWDs) can spin continuously for an hour without a vertigo perception.This unique long term vestibular system stimulation presents a potential human model to clarify the cortical networks underlying the resistance against vertigo. This study, therefore, aimed to investigate the potential structural cortical plasticity in SWDs. Magnetic resonance imaging (MRI) of 10 SWDs and 10 controls were obtained, using a 3T scanner. Cortical thickness in the whole cortex was calculated. Results demonstrated significantly thinner cortical areas for SWD subjects compared with the control group in the hubs of the default mode network (DMN), as well as in the motion perception and discrimination areas including the right dorsolateral prefrontal cortex (DLPFC), the right lingual gyrus and the left visual area 5 (V5)/middle temporal (MT) and the left fusiform gyrus. In conclusion, this is the first report that warrants the potential relationship of the motion/body perception related cortical networks and the prolonged term of whirling ability without vertigo or dizziness.

An Integrated Review of Emoticons in Computer-Mediated Communication

Facial expressions constitute a rich source of non-verbal cues in face-to-face communication. They provide interlocutors with resources to express and interpret verbal messages, which may affect their cognitive and emotional processing. Contrarily, computer-mediated communication (CMC), particularly text-based communication, is limited to the use of symbols to convey a message, where facial expressions cannot be transmitted naturally. In this scenario, people use emoticons as paralinguistic cues to convey emotional meaning. Research has shown that emoticons contribute to a greater social presence as a result of the enrichment of text-based communication channels. Additionally, emoticons constitute a valuable resource for language comprehension by providing expressivity to text messages. The latter findings have been supported by studies in neuroscience showing that particular brain regions involved in emotional processing are also activated when people are exposed to emoticons. To reach an integrated understanding of the influence of emoticons in human communication on both socio-cognitive and neural levels, we review the literature on emoticons in three different areas. First, we present relevant literature on emoticons in CMC. Second, we study the influence of emoticons in language comprehension. Finally, we show the incipient research in neuroscience on this topic. This mini review reveals that, while there are plenty of studies on the influence of emoticons in communication from a social psychology perspective, little is known about the neurocognitive basis of the effects of emoticons on communication dynamics.

Sluggishness of Early-Stage Face Processing (N170) Is Correlated with Negative and General Psychiatric Symptoms in Schizophrenia

Patients with schizophrenia consistently exhibit abnormalities in the N170 event-related potential (ERP) component evoked by images of faces. However, the relationship between these face-specific N170 abnormalities in patients with schizophrenia and the clinical characteristics of this disorder has not been elucidated. Here, ERP recordings were conducted for patients with schizophrenia and healthy controls. The amplitude and latency of the N170 component were recorded while participants passively viewed face and non-face (table) images to explore the correlation between face-specific processing and clinical characteristics in schizophrenia. The results provided evidence for a face-specific N170 latency delay in patients with schizophrenia. The N170 latency in patients with schizophrenia was significantly longer than that in healthy controls when images of faces were presented in both upright and inverted orientations. Importantly, the face-related N170 latencies of the left temporo-occipital electrodes (P7 and PO7) were positively correlated with both negative and general psychiatric symptoms in these patients. The N170 amplitudes were weaker in patients than in controls for inverted images of both faces and non-faces (tables), with a left-hemisphere dominance. The face inversion effect (FIE), meaning the difference in N170 amplitude between upright and inverted faces, was absent in patients with schizophrenia, suggesting an abnormality of holistic face processing. Together, these results revealed a marked symptom-relevant neural delay associated with face-specific processing in patients with schizophrenia, providing additional evidence to support the demyelination hypothesis of schizophrenia.

Rapid gamma oscillations in the inferior occipital gyrus in response to eyes

Eyes are an indispensable communication medium for human social interactions. Although previous neuroscientific evidence suggests the activation of the inferior occipital gyrus (IOG) during eye processing, the temporal profile of this activation remains unclear. To investigate this issue, we analyzed intracranial electroencephalograms of the IOG during the presentation of eyes and mosaics, in either averted or straight directions. Time–frequency statistical parametric mapping analyses revealed greater gamma-band activation in the right IOG beginning at 114 ms in response to eyes relative to mosaics, irrespective of their averted or straight direction. These results suggest that gamma oscillations in the right IOG are involved in the early stages of eye processing, such as eye detection.

Neural Correlates of Attention to Human-Made Sounds: An ERP Study

Previous neuroimaging and electrophysiological studies have suggested that human made sounds are processed differently from non-human made sounds. Multiple groups have suggested that voices might be processed as “special,” much like faces. Although previous literature has explored neural correlates of voice perception under varying task demands, few studies have examined electrophysiological correlates of attention while directly comparing human made and non-human made sounds. In the present study, we used event-related potentials (ERPs) to compare attention to human versus non-human made sounds in an oddball paradigm. ERP components of interest were the P300, and fronto-temporal positivity to voices (FTVP), which has been reported in previous investigations of voice versus non-voice stimuli. We found that participants who heard human made sounds as “target” or infrequent stimuli had significantly larger FTPV amplitude, shorter FTPV latency, and larger P300 amplitude than those who heard non-human sounds as “target” stimuli. Our results are in concordance with previous findings that human-made and non-human made sounds are processed differently, and expand upon previous literature by demonstrating increased attention to human versus non-human made sounds, even when the non-human made sounds are ones that require immediate attention in daily life (e.g. a car horn). Heightened attention to human-made sounds is important theoretically and has potential for application in tests of social interest in populations with autism.

Representational Dynamics of Facial Viewpoint Encoding

Faces provide a wealth of information, including the identity of the seen person and social cues, such as the direction of gaze. Crucially, different aspects of face processing require distinct forms of information encoding. Another person's attentional focus can be derived based on a view-dependent code. In contrast, identification benefits from invariance across all viewpoints. Different cortical areas have been suggested to subserve these distinct functions. However, little is known about the temporal aspects of differential viewpoint encoding in the human brain. Here, we combine EEG with multivariate data analyses to resolve the dynamics of face processing with high temporal resolution. This revealed a distinct sequence of viewpoint encoding. Head orientations were encoded first, starting after around 60 msec of processing. Shortly afterward, peaking around 115 msec after stimulus onset, a different encoding scheme emerged. At this latency, mirror-symmetric viewing angles elicited highly similar cortical responses. Finally, about 280 msec after visual onset, EEG response patterns demonstrated a considerable degree of viewpoint invariance across all viewpoints tested, with the noteworthy exception of the front-facing view. Taken together, our results indicate that the processing of facial viewpoints follows a temporal sequence of encoding schemes, potentially mirroring different levels of computational complexity.

Category-Selectivity in Human Visual Cortex Follows Cortical Topology: A Grouped icEEG Study

Neuroimaging studies suggest that category-selective regions in higher-order visual cortex are topologically organized around specific anatomical landmarks: the mid-fusiform sulcus (MFS) in the ventral temporal cortex (VTC) and lateral occipital sulcus (LOS) in the lateral occipital cortex (LOC). To derive precise structure-function maps from direct neural signals, we collected intracranial EEG (icEEG) recordings in a large human cohort (n = 26) undergoing implantation of subdural electrodes. A surface-based approach to grouped icEEG analysis was used to overcome challenges from sparse electrode coverage within subjects and variable cortical anatomy across subjects. The topology of category-selectivity in bilateral VTC and LOC was assessed for five classes of visual stimuli-faces, animate non-face (animals/body-parts), places, tools, and words-using correlational and linear mixed effects analyses. In the LOC, selectivity for living (faces and animate non-face) and non-living (places and tools) classes was arranged in a ventral-to-dorsal axis along the LOS. In the VTC, selectivity for living and non-living stimuli was arranged in a latero-medial axis along the MFS. Written word-selectivity was reliably localized to the intersection of the left MFS and the occipito-temporal sulcus. These findings provide direct electrophysiological evidence for topological information structuring of functional representations within higher-order visual cortex.

Electrophysiological correlates of face-evoked person knowledge

Face recognition includes identifying a face as perceptually familiar and recollecting biographical information, or person-knowledge, associated with the face. The majority of studies examining the neural basis of face recognition have confounded these stages by comparing brain responses evoked by novel and perceptually familiar famous faces. Here, we recorded EEG in two tasks in which subjects viewed two sets of faces that were equally perceptually familiar, but which had differing levels of associated person-knowledge. Our results dissociated the effects of person-knowledge from perceptual familiarity. Faces with associated biographical information elicited a larger ∼600ms centroparietal positivity in both a passive viewing task in which subjects viewed faces without explicitly responding, and an active question-answering task in which subjects indicated whether or not they knew particular facts about the faces. In the question task only, person-knowledge was associated with a negative ERP difference over right posterior scalp over the 170-450ms interval which appeared again at long latency (>900ms).

Abnormalities of Inter- and Intra-Hemispheric Functional Connectivity in Autism Spectrum Disorders: A Study Using the Autism Brain Imaging Data Exchange Database

Recently, the Autism Brain Imaging Data Exchange (ABIDE) project revealed decreased functional connectivity in individuals with Autism Spectrum Disorders (ASD) relative to the typically developing controls (TDCs). However, it is still questionable whether the source of functional under-connectivity in subjects with ASD is equally contributed by the ipsilateral and contralateral parts of the brain. In this study, we decomposed the inter- and intra-hemispheric regions and compared the functional connectivity density (FCD) between 458 subjects with ASD and 517 TDCs from the ABIDE database. We quantified the inter- and intra-hemispheric FCDs in the brain by counting the number of functional connectivity with all voxels in the opposite and same hemispheric brain regions, respectively. Relative to TDCs, both inter- and intra-hemispheric FCDs in the posterior cingulate cortex, lingual/parahippocampal gyrus, and postcentral gyrus were significantly decreased in subjects with ASD. Moreover, in the ASD group, the restricted and repetitive behavior subscore of the Autism Diagnostic Observation Schedule (ADOS-RRB) score showed significant negative correlations with the average inter-hemispheric FCD and contralateral FCD in the lingual/parahippocampal gyrus cluster. Also, the ADOS-RRB score showed significant negative correlations with the average contralateral FCD in the default mode network regions such as the posterior cingulate cortex and precuneus. Taken together, our findings imply that a deficit of non-social functioning processing in ASD such as restricted and repetitive behaviors and sensory hypersensitivity could be determined via both inter- and intra-hemispheric functional disconnections.

The "parts and wholes" of face recognition: A review of the literature

It has been claimed that faces are recognized as a "whole" rather than by the recognition of individual parts. In a paper published in the Quarterly Journal of Experimental Psychology in 1993, Martha Farah and I attempted to operationalize the holistic claim using the part/whole task. In this task, participants studied a face and then their memory presented in isolation and in the whole face. Consistent with the holistic view, recognition of the part was superior when tested in the whole-face condition compared to when it was tested in isolation. The "whole face" or holistic advantage was not found for faces that were inverted, or scrambled, nor for non-face objects, suggesting that holistic encoding was specific to normal, intact faces. In this paper, we reflect on the part/whole paradigm and how it has contributed to our understanding of what it means to recognize a face as a "whole" stimulus. We describe the value of part/whole task for developing theories of holistic and non-holistic recognition of faces and objects. We discuss the research that has probed the neural substrates of holistic processing in healthy adults and people with prosopagnosia and autism. Finally, we examine how experience shapes holistic face recognition in children and recognition of own- and other-race faces in adults. The goal of this article is to summarize the research on the part/whole task and speculate on how it has informed our understanding of holistic face processing.

Implicit Processing of the Eyes and Mouth: Evidence from Human Electrophysiology

The current study examined the time course of implicit processing of distinct facial features and the associate event-related potential (ERP) components. To this end, we used a masked priming paradigm to investigate implicit processing of the eyes and mouth in upright and inverted faces, using a prime duration of 33 ms. Two types of prime-target pairs were used: 1. congruent (e.g., open eyes only in both prime and target or open mouth only in both prime and target); 2. incongruent (e.g., open mouth only in prime and open eyes only in target or open eyes only in prime and open mouth only in target). The identity of the faces changed between prime and target. Participants pressed a button when the target face had the eyes open and another button when the target face had the mouth open. The behavioral results showed faster RTs for the eyes in upright faces than the eyes in inverted faces, the mouth in upright and inverted faces. Moreover they also revealed a congruent priming effect for the mouth in upright faces. The ERP findings showed a face orientation effect across all ERP components studied (P1, N1, N170, P2, N2, P3) starting at about 80 ms, and a congruency/priming effect on late components (P2, N2, P3), starting at about 150 ms. Crucially, the results showed that the orientation effect was driven by the eye region (N170, P2) and that the congruency effect started earlier (P2) for the eyes than for the mouth (N2). These findings mark the time course of the processing of internal facial features and provide further evidence that the eyes are automatically processed and that they are very salient facial features that strongly affect the amplitude, latency, and distribution of neural responses to faces.

Neuronal codes for visual perception and memory

In this review, I describe and contrast the representation of stimuli in visual cortical areas and in the medial temporal lobe (MTL). While cortex is characterized by a distributed and implicit coding that is optimal for recognition and storage of semantic information, the MTL shows a much sparser and explicit coding of specific concepts that is ideal for episodic memory. I will describe the main characteristics of the coding in the MTL by the so-called concept cells and will then propose a model of the formation and recall of episodic memory based on partially overlapping assemblies.

Eye gaze and head orientation modulate the inhibition of return for faces

The present study used an inhibition of return (IOR) spatial cueing paradigm to examine how gaze direction and head orientation modulate attention capture for human faces. Target response time (RT) was measured after the presentation of a peripheral cue, which was either a face (with front-facing or averted gaze, in either frontal head view or averted head view) or a house (control). Participants fixated on a centered cross at all times and responded via button press to a peripheral target after a variable stimulus onset asynchrony (SOA) from the stimulus cue. At the shortest SOA (150 ms), RTs were shorter for faces than houses, independent of an IOR response, suggesting a cue-based RT advantage elicited by faces. At the longest SOA (2,400 ms), a larger IOR magnitude was found for faces compared to houses. Both the cue-based RT advantage and later IOR responses were modulated by gaze-head congruency; these effects were strongest for frontal gaze faces in frontal head view, and for averted gaze faces in averted head view. Importantly, participants were not given any specific information regarding the stimuli, nor were they told the true purpose of the study. These findings indicate that the congruent combination of head and gaze direction influence the exogenous attention capture of faces during inhibition of return.

Spatial frequency selection and integration of global and local information in visual processing: A selective review and tribute to Shlomo Bentin

Previous research has suggested a relationship between processing lower versus higher spatial frequencies (SFs) and global/local perception, respectively. Here we honor Shlomo Bentin by reviewing the work we conducted with him regarding this issue. This work was aimed at investigating the mechanisms by which selective attention to spatial frequency (SF) mediates global and local perception in general and how these perceptual levels are integrated with the shapes that define them. The experiments demonstrate that attention to global and local aspects of a hierarchical display biases the flexible selection of relatively lower and relatively higher SFs during image processing. Additionally, attentional selection of SF allows for the shapes in a hierarchical display to be integrated with the level (global/local) at which they occur. The studies reviewed here provide strong evidence that the flexible, top-down selection of low-level SF channels mediates the perception of global and local elements of visual displays. The studies also support a hemisphere asymmetry in this process, with right hemisphere functions biased toward global perception and left hemisphere functions biased toward local.

The 170ms Response to Faces as Measured by MEG (M170) Is Consistently Altered in Congenital Prosopagnosia

Modularity of face processing is still a controversial issue. Congenital prosopagnosia (cPA), a selective and lifelong impairment in familiar face recognition without evidence of an acquired cerebral lesion, offers a unique opportunity to support this fundamental hypothesis. However, in spite of the pronounced behavioural impairment, identification of a functionally relevant neural alteration in congenital prosopagnosia by electrophysiogical methods has not been achieved so far. Here we show that persons with congenital prosopagnosia can be distinguished as a group from unimpaired persons using magnetoencephalography. Early face-selective MEG-responses in the range of 140 to 200ms (the M170) showed prolonged latency and decreased amplitude whereas responses to another category (houses) were indistinguishable between subjects with congenital prosopagnosia and unimpaired controls. Latency and amplitude of face-selective EEG responses (the N170) which were simultaneously recorded were statistically indistinguishable between subjects with cPA and healthy controls which resolves heterogeneous and partly conflicting results from existing studies. The complementary analysis of categorical differences (evoked activity to faces minus evoked activity to houses) revealed that the early part of the 170ms response to faces is altered in subjects with cPA. This finding can be adequately explained in a common framework of holistic and part-based face processing. Whereas a significant brain-behaviour correlation of face recognition performance and the size of the M170 amplitude is found in controls a corresponding correlation is not seen in subjects with cPA. This indicates functional relevance of the alteration found for the 170ms response to faces in cPA and pinpoints the impairment of face processing to early perceptual stages.

Neural mechanisms underlying conscious and unconscious attentional shifts triggered by eye gaze

Behavioral studies have shown that eye gaze triggers attentional shifts both with and without conscious awareness. However, the neural substrates of conscious and unconscious attentional shifts triggered by eye gaze remain unclear. To investigate this issue, we measured brain activity using event-related functional magnetic resonance imaging while participants observed averted or straight eye-gaze cues presented supraliminally or subliminally in the central visual field and then localized a subsequent target in the peripheral visual field. Reaction times for localizing the targets were shorter under both supraliminal and subliminal conditions when eye-gaze cues were directionally congruent with the target locations than when they were directionally neutral. Conjunction analyses revealed that a bilateral cortical network, including the middle temporal gyri, inferior parietal lobules, anterior cingulate cortices, and superior and middle frontal gyri, was activated more in response to averted eyes than to straight eyes under both supraliminal and subliminal conditions. Interaction analyses revealed that the right inferior parietal lobule was specifically active when participants viewed averted eyes relative to straight eyes under the supraliminal condition; the bilateral subcortical regions, including the superior colliculus and amygdala, and the middle temporal and inferior frontal gyri in the right hemisphere were activated in response to averted versus straight eyes under the subliminal condition. These results suggest commonalities and differences in the neural mechanisms underlying conscious and unconscious attentional shifts triggered by eye gaze.

Functional asymmetry between the left and right human fusiform gyrus explored through electrical brain stimulation

The ventral temporal cortex (VTC) contains several areas with selective responses to words, numbers, faces, and objects as demonstrated by numerous human and primate imaging and electrophysiological studies. Our recent work using electrocorticography (ECoG) confirmed the presence of face-selective neuronal populations in the human fusiform gyrus (FG) in patients implanted with intracranial electrodes in either the left or right hemisphere. Electrical brain stimulation (EBS) disrupted the conscious perception of faces only when it was delivered in the right, but not left, FG. In contrast to our previous findings, here we report both negative and positive EBS effects in right and left FG, respectively. The presence of right hemisphere language dominance in the first, and strong left-handedness and poor language processing performance in the second case, provide indirect clues about the functional architecture of the human VTC in relation to hemispheric asymmetries in language processing and handedness.

Corresponding ECoG and fMRI category-selective signals in human ventral temporal cortex

Functional magnetic resonance imaging (fMRI) and electrocorticography (ECoG) research have been influential in revealing the functional characteristics of category-selective responses in human ventral temporal cortex (VTC). One important, but unanswered, question is how these two types of measurements might be related with respect to the VTC. Here we examined which components of the ECoG signal correspond to the fMRI response by using a rare opportunity to measure both fMRI and ECoG responses from the same individuals to images of exemplars of various categories including faces, limbs, cars and houses. Our data reveal three key findings. First, we discovered that the coupling between fMRI and ECoG responses is frequency and time dependent. The strongest and most sustained correlation is observed between fMRI and high frequency broadband (HFB) ECoG responses (30-160 hz). In contrast, the correlation between fMRI and ECoG signals in lower frequency bands is temporally transient, where the correlation is initially positive, but then tapers off or becomes negative. Second, we find that the strong and positive correlation between fMRI and ECoG signals in all frequency bands emerges rapidly around 100 ms after stimulus onset, together with the onset of the first stimulus-driven neural signals in VTC. Third, we find that the spatial topology and representational structure of category-selectivity in VTC reflected in ECoG HFB responses mirrors the topology and structure observed with fMRI. These findings of a strong and rapid coupling between fMRI and HFB responses validate fMRI measurements of functional selectivity with recordings of direct neural activity and suggest that fMRI category-selective signals in VTC are associated with feed-forward neural processing.

Development of grouped icEEG for the study of cognitive processing

Invasive intracranial EEG (icEEG) offers a unique opportunity to study human cognitive networks at an unmatched spatiotemporal resolution. To date, the contributions of icEEG have been limited to the individual-level analyses or cohorts whose data are not integrated in any way. Here we discuss how grouped approaches to icEEG overcome challenges related to sparse-sampling, correct for individual variations in response and provide statistically valid models of brain activity in a population. By the generation of whole-brain activity maps, grouped icEEG enables the study of intra and interregional dynamics between distributed cortical substrates exhibiting task-dependent activity. In this fashion, grouped icEEG analyses can provide significant advances in understanding the mechanisms by which cortical networks give rise to cognitive functions.

Brain Signals of Face Processing as Revealed by Event-Related Potentials

We analyze the functional significance of different event-related potentials (ERPs) as electrophysiological indices of face perception and face recognition, according to cognitive and neurofunctional models of face processing. Initially, the processing of faces seems to be supported by early extrastriate occipital cortices and revealed by modulations of the occipital P1. This early response is thought to reflect the detection of certain primary structural aspects indicating the presence grosso modo of a face within the visual field. The posterior-temporal N170 is more sensitive to the detection of faces as complex-structured stimuli and, therefore, to the presence of its distinctive organizational characteristics prior to within-category identification. In turn, the relatively late and probably more rostrally generated N250r and N400-like responses might respectively indicate processes of access and retrieval of face-related information, which is stored in long-term memory (LTM). New methods of analysis of electrophysiological and neuroanatomical data, namely, dynamic causal modeling, single-trial and time-frequency analyses, are highly recommended to advance in the knowledge of those brain mechanisms concerning face processing.

Dynamic encoding of face information in the human fusiform gyrus

Humans’ ability to rapidly and accurately detect, identify, and classify faces under variable conditions derives from a network of brain regions highly tuned to face information. The fusiform face area (FFA) is thought to be a computational hub for face processing, however temporal dynamics of face information processing in FFA remains unclear. Here we use multivariate pattern classification to decode the temporal dynamics of expression-invariant face information processing using electrodes placed directly upon FFA in humans. Early FFA activity (50-75 ms) contained information regarding whether participants were viewing a face. Activity between 200-500 ms contained expression-invariant information about which of 70 faces participants were viewing along with the individual differences in facial features and their configurations. Long-lasting (500+ ms) broadband gamma frequency activity predicted task performance. These results elucidate the dynamic computational role FFA plays in multiple face processing stages and indicate what information is used in performing these visual analyses.

Electrical stimulation of the left and right human fusiform gyrus causes different effects in conscious face perception

Neuroimaging and electrophysiological studies across species have confirmed bilateral face-selective responses in the ventral temporal cortex (VTC) and prosopagnosia is reported in patients with lesions in the VTC including the fusiform gyrus (FG). As imaging and electrophysiological studies provide correlative evidence, and brain lesions often comprise both white and gray matter structures beyond the FG, we designed the current study to explore the link between face-related electrophysiological responses in the FG and the causal effects of electrical stimulation of the left or right FG in face perception. We used a combination of electrocorticography (ECoG) and electrical brain stimulation (EBS) in 10 human subjects implanted with intracranial electrodes in either the left (5 participants, 30 FG sites) or right (5 participants, 26 FG sites) hemispheres. We identified FG sites with face-selective ECoG responses, and recorded perceptual reports during EBS of these sites. In line with existing literature, face-selective ECoG responses were present in both left and right FG sites. However, when the same sites were stimulated, we observed a striking difference between hemispheres. Only EBS of the right FG caused changes in the conscious perception of faces, whereas EBS of strongly face-selective regions in the left FG produced non-face-related visual changes, such as phosphenes. This study examines the relationship between correlative versus causal nature of ECoG and EBS, respectively, and provides important insight into the differential roles of the right versus left FG in conscious face perception.

Hemodynamic Response Pattern of Spatial Cueing is Different for Social and Symbolic Cues

Directional social gaze and symbolic arrow cues both serve as spatial cues, causing seemingly reflexive shifts of an observer's attention. However, the underlying neural substrates remain a point at issue. The present study specifically addressed the differences in the activation patterns associated with non-predictive gaze and arrow cues, placing special emphasis on brain regions known to be involved in the processing of social information [superior temporal sulcus (STS), fusiform gyrus (FFG)]. Additionally, the functional connectivity of these brain regions with other areas involved in gaze processing and spatial attention was investigated. Results indicate that gaze and arrow cues recruit several brain regions differently, with gaze cues increasing activation in occipito-temporal regions and arrow cues increasing activation in occipito-parietal regions. Specifically, gaze cues in contrast to arrow cues enhanced activation in the FFG and the STS. Functional connectivity analysis revealed that during gaze cueing the STS was more strongly connected to the intraparietal sulcus (IPS) and the frontal eye fields, whereas the FFG was more strongly connected to the IPS and the amygdala.

Facial expression recognition takes longer in the posterior superior temporal sulcus than in the occipital face area

Neuroimaging studies have identified a face-selective region in the right posterior superior temporal sulcus (rpSTS) that responds more strongly during facial expression recognition tasks than during facial identity recognition tasks, but precisely when the rpSTS begins to causally contribute to expression recognition is unclear. The present study addressed this issue using transcranial magnetic stimulation (TMS). In Experiment 1, repetitive TMS delivered over the rpSTS of human participants, at a frequency of 10 Hz for 500 ms, selectively impaired a facial expression task but had no effect on a matched facial identity task. In Experiment 2, participants performed the expression task only while double-pulse TMS (dTMS) was delivered over the rpSTS or over the right occipital face area (rOFA), a face-selective region in lateral occipital cortex, at different latencies up to 210 ms after stimulus onset. Task performance was selectively impaired when dTMS was delivered over the rpSTS at 60-100 ms and 100-140 ms. dTMS delivered over the rOFA impaired task performance at 60-100 ms only. These results demonstrate that the rpSTS causally contributes to expression recognition and that it does so over a longer time-scale than the rOFA. This difference in the length of the TMS induced impairment between the rpSTS and the rOFA suggests that the neural computations that contribute to facial expression recognition in each region are functionally distinct.

Face, eye, and body selective responses in fusiform gyrus and adjacent cortex: an intracranial EEG study

Functional MRI (fMRI) studies have investigated the degree to which processing of whole faces, face-parts, and bodies are differentially localized within the fusiform gyrus and adjacent ventral occipitotemporal cortex. While some studies have emphasized the spatial differentiation of processing into discrete areas, others have emphasized the overlap of processing and the importance of distributed patterns of activity. Intracranial EEG (iEEG) recorded from subdural electrodes provides excellent temporal and spatial resolution of local neural activity, and thus provides an alternative method to fMRI for studying differences and commonalities in face and body processing. In this study we recorded iEEG from 12 patients while they viewed images of novel faces, isolated eyes, headless bodies, and flowers. Event-related potential analysis identified 69 occipitotemporal sites at which there was a face-, eye-, or body-selective response when contrasted to flowers. However, when comparing faces, eyes, and bodies to each other at these sites, we identified only 3 face-specific, 13 eye-specific, and 1 body-specific electrodes. Thus, at the majority of sites, faces, eyes, and bodies evoked similar responses. However, we identified ten locations at which the amplitude of the responses spatially varied across adjacent electrodes, indicating that the configuration of current sources and sinks were different for faces, eyes, and bodies. Our results also demonstrate that eye-sensitive regions are more abundant and more purely selective than face- or body-sensitive regions, particularly in lateral occipitotemporal cortex.

The functional architecture of the ventral temporal cortex and its role in categorization

Visual categorization is thought to occur in the human ventral temporal cortex (VTC), but how this categorization is achieved is still largely unknown. In this Review, we consider the computations and representations that are necessary for categorization and examine how the microanatomical and macroanatomical layout of the VTC might optimize them to achieve rapid and flexible visual categorization. We propose that efficient categorization is achieved by organizing representations in a nested spatial hierarchy in the VTC. This spatial hierarchy serves as a neural infrastructure for the representational hierarchy of visual information in the VTC and thereby enables flexible access to category information at several levels of abstraction.

Selective attention modulates high-frequency activity in the face-processing network

Face processing depends on the orchestrated activity of a large-scale neuronal network. Its activity can be modulated by attention as a function of task demands. However, it remains largely unknown whether voluntary, endogenous attention and reflexive, exogenous attention to facial expressions equally affect all regions of the face-processing network, and whether such effects primarily modify the strength of the neuronal response, the latency, the duration, or the spectral characteristics. We exploited the good temporal and spatial resolution of intracranial electroencephalography (iEEG) and recorded from depth electrodes to uncover the fast dynamics of emotional face processing. We investigated frequency-specific responses and event-related potentials (ERP) in the ventral occipito-temporal cortex (VOTC), ventral temporal cortex (VTC), anterior insula, orbitofrontal cortex (OFC), and amygdala when facial expressions were task-relevant or task-irrelevant. All investigated regions of interest (ROI) were clearly modulated by task demands and exhibited stronger changes in stimulus-induced gamma band activity (50-150 Hz) when facial expressions were task-relevant. Observed latencies demonstrate that the activation is temporally coordinated across the network, rather than serially proceeding along a processing hierarchy. Early and sustained responses to task-relevant faces in VOTC and VTC corroborate their role for the core system of face processing, but they also occurred in the anterior insula. Strong attentional modulation in the OFC and amygdala (300 msec) suggests that the extended system of the face-processing network is only recruited if the task demands active face processing. Contrary to our expectation, we rarely observed differences between fearful and neutral faces. Our results demonstrate that activity in the face-processing network is susceptible to the deployment of selective attention. Moreover, we show that endogenous attention operates along the whole face-processing network, and that these effects are reflected in frequency-specific changes in the gamma band.

Neuromagnetic vistas into typical and atypical development of frontal lobe functions

The frontal lobes are involved in many higher-order cognitive functions such as social cognition executive functions and language and speech. These functions are complex and follow a prolonged developmental course from childhood through to early adulthood. Magnetoencephalography (MEG) is ideal for the study of development of these functions, due to its combination of temporal and spatial resolution which allows the determination of age-related changes in both neural timing and location. There are several challenges for MEG developmental studies: to design tasks appropriate to capture the neurodevelopmental trajectory of these cognitive functions, and to develop appropriate analysis strategies to capture various aspects of neuromagnetic frontal lobe activity. Here, we review our MEG research on social and executive functions, and speech in typically developing children and in two clinical groups – children with autism spectrum disorder and children born very preterm. The studies include facial emotional processing, inhibition, visual short-term memory, speech production, and resting-state networks. We present data from event-related analyses as well as on oscillations and connectivity analyses and review their contributions to understanding frontal lobe cognitive development. We also discuss the challenges of testing young children in the MEG and the development of age-appropriate technologies and paradigms.

Beyond the FFA: The role of the ventral anterior temporal lobes in face processing

Extensive research has supported the existence of a specialized face-processing network that is distinct from the visual processing areas used for general object recognition. The majority of this work has been aimed at characterizing the response properties of the fusiform face area (FFA) and the occipital face area (OFA), which together are thought to constitute the core network of brain areas responsible for facial identification. Although accruing evidence has shown that face-selective patches in the ventral anterior temporal lobes (vATLs) are interconnected with the FFA and OFA, and that they play a role in facial identification, the relative contribution of these brain areas to the core face-processing network has remained unarticulated. Here we review recent research critically implicating the vATLs in face perception and memory. We propose that current models of face processing should be revised such that the ventral anterior temporal lobes serve a centralized role in the visual face-processing network. We speculate that a hierarchically organized system of face processing areas extends bilaterally from the inferior occipital gyri to the vATLs, with facial representations becoming increasingly complex and abstracted from low-level perceptual features as they move forward along this network. The anterior temporal face areas may serve as the apex of this hierarchy, instantiating the final stages of face recognition. We further argue that the anterior temporal face areas are ideally suited to serve as an interface between face perception and face memory, linking perceptual representations of individual identity with person-specific semantic knowledge.

Processing of configural and componential information in face-selective cortical areas

We investigated how face-selective cortical areas process configural and componential face information and how race of faces may influence these processes. Participants saw blurred (preserving configural information), scrambled (preserving componential information), and whole faces during fMRI scan, and performed a post-scan face recognition task using blurred or scrambled faces. The fusiform face area (FFA) showed stronger activation to blurred than to scrambled faces, and equivalent responses to blurred and whole faces. The occipital face area (OFA) showed stronger activation to whole than to blurred faces, which elicited similar responses to scrambled faces. Therefore, the FFA may be more tuned to process configural than componential information, whereas the OFA similarly participates in perception of both. Differences in recognizing own- and other-race blurred faces were correlated with differences in FFA activation to those faces, suggesting that configural processing within the FFA may underlie the other-race effect in face recognition.

Neural dynamics underlying target detection in the human brain

Sensory signals must be interpreted in the context of goals and tasks. To detect a target in an image, the brain compares input signals and goals to elicit the correct behavior. We examined how target detection modulates visual recognition signals by recording intracranial field potential responses from 776 electrodes in 10 epileptic human subjects. We observed reliable differences in the physiological responses to stimuli when a cued target was present versus absent. Goal-related modulation was particularly strong in the inferior temporal and fusiform gyri, two areas important for object recognition. Target modulation started after 250 ms post stimulus, considerably after the onset of visual recognition signals. While broadband signals exhibited increased or decreased power, gamma frequency power showed predominantly increases during target presence. These observations support models where task goals interact with sensory inputs via top-down signals that influence the highest echelons of visual processing after the onset of selective responses.

Localizing evoked and induced responses to faces using magnetoencephalography

A rich pattern of responses in frequency, time and space are known to be generated in the visual cortex in response to faces. Recently, a number of studies have used magnetoencephalography (MEG) to try to record these responses non-invasively – in many cases using source analysis techniques based on the beamforming method. Here we sought both to characterize best practice for measuring face-specific responses using MEG beamforming, and to determine whether the results produced by the beamformer match evidence from other modalities. We measured activity to visual presentation of face stimuli and phase-scrambled control stimuli, and performed source analyses of both induced and evoked responses using Synthetic Aperture Magnetometry. We localized the gamma-band response to bilateral lateral occipital cortex, and both the gamma-band response and the M170-evoked response to the right fusiform gyrus. Differences in the gamma-band response between faces and scrambled stimuli were confined to the frequency range 50–90 Hz; gamma-band activity at higher frequencies did not differ between the two stimulus categories. We additionally identified a component of the M220-evoked response – localized to the parieto-occipital sulcus – which was enhanced for scrambled vs. unscrambled faces. These findings help to establish that MEG beamforming can localize face-specific responses in time, frequency and space with good accuracy (when validated against established findings from functional magnetic resonance imaging and intracranial recordings), as well as contributing to the establishment of best methodological practice for the use of the beamformer method to measure face-specific responses.

Replay of very early encoding representations during recollection

Long-term memories are linked to cortical representations of perceived events, but it is unclear which types of representations can later be recollected. Using magnetoencephalography-based decoding, we examined which brain activity patterns elicited during encoding are later replayed during recollection in the human brain. The results show that the recollection of images depicting faces and scenes is associated with a replay of neural representations that are formed at very early (180 ms) stages of encoding. This replay occurs quite rapidly, ~500 ms after the onset of a cue that prompts recollection and correlates with source memory accuracy. Therefore, long-term memories are rapidly replayed during recollection and involve representations that were formed at very early stages of encoding. These findings indicate that very early representational information can be preserved in the memory engram and can be faithfully and rapidly reinstated during recollection. These novel insights into the nature of the memory engram provide constraints for mechanistic models of long-term memory function.

Resolving human object recognition in space and time

A comprehensive picture of object processing in the human brain requires combining both spatial and temporal information about brain activity. Here we acquired human magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI) responses to 92 object images. Multivariate pattern classification applied to MEG revealed the time course of object processing: whereas individual images were discriminated by visual representations early, ordinate and superordinate category levels emerged relatively late. Using representational similarity analysis, we combined human fMRI and MEG to show content-specific correspondence between early MEG responses and primary visual cortex (V1), and later MEG responses and inferior temporal (IT) cortex. We identified transient and persistent neural activities during object processing with sources in V1 and IT. Finally, we correlated human MEG signals to single-unit responses in monkey IT. Together, our findings provide an integrated space- and time-resolved view of human object categorization during the first few hundred milliseconds of vision.

The building blocks of social communication

In the present review, social communication will be discussed in the context of social cognition, and cold and hot cognition. The review presents research on prosody, processing of faces, multimodal processing of voice and face, and the impact of emotion on constructing semantic meaning. Since the focus of this mini review is on brain processes involved in these cognitive functions, the bulk of evidence presented will be from event related potential (ERP) studies as this methodology offers the best temporal resolution of cognitive events under study. The argument is made that social communication is accomplished via fast acting sensory processes and later, top down processes. Future directions both in terms of methodology and research questions are also discussed.

Alternating zones selective to faces and written words in the human ventral occipitotemporal cortex

Recognition of faces and written words is associated with category-specific brain activation in the ventral occipitotemporal cortex (vOT). However, topological and functional relationships between face-selective and word-selective vOT regions remain unclear. In this study, we collected data from patients with intractable epilepsy who underwent high-density recording of surface field potentials in the vOT. "Faces" and "letterstrings" induced outstanding category-selective responses among the 24 visual categories tested, particularly in high-γ band powers. Strikingly, within-hemispheric analysis revealed alternation of face-selective and letterstring-selective zones within the vOT. Two distinct face-selective zones located anterior and posterior portions of the mid-fusiform sulcus whereas letterstring-selective zones alternated between and outside of these 2 face-selective zones. Further, a classification analysis indicated that activity patterns of these zones mostly represent dedicated categories. Functional connectivity analysis using Granger causality indicated asymmetrically directed causal influences from face-selective to letterstring-selective regions. These results challenge the prevailing view that different categories are represented in distinct contiguous regions in the vOT.