Perception and Memory Reinstatement Engage Overlapping Face-Selective Regions within Human Ventral Temporal Cortex

Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions specialized for processing specific sensory stimuli are reactivated to support content-specific retrieval. Recently, several studies have emphasized transformations in the spatial organization of these reinstated activity patterns. Specifically, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with the activity observed during perception. However, it is not clear that such transformations occur universally, with inconsistent evidence for other important stimulus categories, particularly faces. One challenge in addressing this question is the careful delineation of face-selective cortices, which are interdigitated with other selective regions, in configurations that spatially differ across individuals. Therefore, we conducted a multisession neuroimaging study to first carefully map individual participants' (nine males and seven females) face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the activity patterns within these regions during face memory encoding and retrieval. While face-selective regions were expectedly engaged during face perception at encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions, but did not show any consistent direction of spatial transformation (e.g., anteriorization). We also report on unique human intracranial recordings from VTC under the same experimental conditions. These findings highlight the importance of considering the complex configuration of category-selective cortex in elucidating principles shaping the neural transformations that occur from perception to memory.

Developing cortex is functionally pluripotent: Evidence from blindness

How rigidly does innate architecture constrain function of developing cortex? What is the contribution of early experience? We review insights into these questions from visual cortex function in people born blind. In blindness, occipital cortices are active during auditory and tactile tasks. What 'cross-modal' plasticity tells us about cortical flexibility is debated. On the one hand, visual networks of blind people respond to higher cognitive information, such as sentence grammar, suggesting drastic repurposing. On the other, in line with 'metamodal' accounts, sighted and blind populations show shared domain preferences in ventral occipito-temporal cortex (vOTC), suggesting visual areas switch input modality but perform the same or similar perceptual functions (e.g., face recognition) in blindness. Here we bring these disparate literatures together, reviewing and synthesizing evidence that speaks to whether visual cortices have similar or different functions in blind and sighted people. Together, the evidence suggests that in blindness, visual cortices are incorporated into higher-cognitive (e.g., fronto-parietal) networks, which are a major source long-range input to the visual system. We propose the connectivity-constrained experience-dependent account. Functional development is constrained by innate anatomical connectivity, experience and behavioral needs. Infant cortex is pluripotent, the same anatomical constraints develop into different functional outcomes.

Sentence-level embeddings reveal dissociable word- and sentence-level cortical representation across coarse- and fine-grained levels of meaning

In this large-sample (N = 64) fMRI study, sentence embeddings (text-embedding-ada-002, OpenAI) and representational similarity analysis were used to contrast sentence-level and word-level semantic representation. Overall, sentence-level information resulted in a 20-25 % increase in the model's ability to captures neural representation when compared to word-level only information (word-order scrambled embeddings). This increase was relatively undifferentiated across the cortex. However, when coarse-grained (across thematic category) and fine-grained (within thematic category) combinatorial meaning were separately assessed, word- and sentence-level representations were seen to strongly dissociate across the cortex and to do so differently as a function of grain. Coarse-grained sentence-level representations were evident in occipitotemporal, ventral temporal and medial prefrontal cortex, while fine-grained differences were seen in lateral prefrontal and parietal cortex, middle temporal gyrus, the precuneus, and medial prefrontal cortex. This result indicates dissociable cortical substrates underly single concept versus combinatorial meaning and that different cortical regions specialise for fine- and coarse-grained meaning.

Neural specialization for 'visual' concepts emerges in the absence of vision

Vision provides a key source of information about many concepts, including 'living things' (e.g., tiger) and visual events (e.g., sparkle). According to a prominent theoretical framework, neural specialization for different conceptual categories is shaped by sensory features, e.g., living things are neurally dissociable from navigable places because living things concepts depend more on visual features. We tested this framework by comparing the neural basis of 'visual' concepts across sighted (n=22) and congenitally blind (n=21) adults. Participants judged the similarity of words varying in their reliance on vision while undergoing fMRI. We compared neural responses to living things nouns (birds, mammals) and place nouns (natural, manmade). In addition, we compared visual event verbs (e.g., 'sparkle') to non-visual events (sound emission, hand motion, mouth motion). People born blind exhibited distinctive univariate and multivariate responses to living things in a temporo-parietal semantic network activated by nouns, including the precuneus (PC). To our knowledge, this is the first demonstration that neural selectivity for living things does not require vision. We additionally observed preserved neural signatures of 'visual' light events in the left middle temporal gyrus (LMTG+). Across a wide range of semantic types, neural representations of sensory concepts develop independent of sensory experience.

Representation of event and object concepts in ventral anterior temporal lobe and angular gyrus

Semantic knowledge includes understanding of objects and their features and also understanding of the characteristics of events. The hub-and-spoke theory holds that these conceptual representations rely on multiple information sources that are integrated in a central hub in the ventral anterior temporal lobes. The dual-hub theory expands this framework with the claim that the ventral anterior temporal lobe hub is specialized for object representation, while a second hub in angular gyrus is specialized for event representation. To test these ideas, we used representational similarity analysis, univariate and psychophysiological interaction analyses of fMRI data collected while participants processed object and event concepts (e.g. "an apple," "a wedding") presented as images and written words. Representational similarity analysis showed that angular gyrus encoded event concept similarity more than object similarity, although the left angular gyrus also encoded object similarity. Bilateral ventral anterior temporal lobes encoded both object and event concept structure, and left ventral anterior temporal lobe exhibited stronger coding for events. Psychophysiological interaction analysis revealed greater connectivity between left ventral anterior temporal lobe and right pMTG, and between right angular gyrus and bilateral ITG and middle occipital gyrus, for event concepts compared to object concepts. These findings support the specialization of angular gyrus for event semantics, though with some involvement in object coding, but do not support ventral anterior temporal lobe specialization for object concepts.

Individual-specific memory reinstatement patterns within human face-selective cortex

Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions involved in the sensory processing of prior events are reactivated to support this perception of the past. Recently, several studies have emphasized potential transformations in the spatial organization of reinstated activity patterns. In particular, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with those during perception. However, it is not clear that such transformations occur universally, with evidence lacking for other important stimulus categories, particularly faces. Critical to addressing these questions, and to studies of reinstatement more broadly, is the growing importance of considering meaningful variations in the organization of sensory systems across individuals. Therefore, we conducted a multi-session neuroimaging study to first carefully map individual participants face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the correspondence of activity patterns during face memory encoding and retrieval. Our results showed distinct configurations of face-selective regions within the VTC across individuals. While a significant degree of overlap was observed between face perception and memory encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions. Importantly, these activity patterns were consistently tied to individual-specific neural substrates, but did not show any consistent direction of spatial transformation (e.g., anteriorization). To provide further insight to these findings, we also report on unique human intracranial recordings from VTC under the same experimental conditions. Our findings highlight the importance of considering individual variations in functional neuroanatomy in the context of assessing the nature of cortical reinstatement. Consideration of such factors will be important for establishing general principles shaping the neural transformations that occur from perception to memory.

Similar representation of names and faces in the network for person perception

Person-knowledge encompasses the diverse types of knowledge we have about other people. This knowledge spans the social, physical, episodic, semantic & nominal information we possess about others and is served by a distributed cortical network including core (perceptual) and extended (non-perceptual) subsystems. Our understanding of this cortical system is tightly linked to the perception of faces and the extent to which cortical knowledge-access processes are independent of perception is unclear. In this study, participants were presented with the written names of famous people and performed ten different semantic access tasks drawn from five cognitive domains (biographic, episodic, nominal, social and physical). We used representational similarity analysis, adapted to investigate network-level representations (NetRSA) to characterise the inter-regional functional coordination within the non-perceptual extended subsystem across access to varied forms of person-knowledge. Results indicate a hierarchical cognitive taxonomy consistent with that seen during face-processing and forming the same three macro-domains: socio-perceptual judgements, episodic-semantic memory and nominal knowledge. The coordination across regions was largely preserved within elements of the extended system associated with internalised cognition but differed in prefrontal regions. Results suggest the elements of the extended system work together in a consistent way to access knowledge when viewing faces and names but that coordination patterns also change as a function of input-processing demands.

Similarity-based fMRI-MEG fusion reveals hierarchical organisation within the brain's semantic system

Our ability to understand and interact with our environment relies upon conceptual knowledge of the meaning of objects. This process is supported by a distributed network of frontal, parietal, and temporal brain regions. Insight into the differential roles of various elements of this system can be inferred from the timing of activation, and here we use similarity-based fMRI-MEG fusion to understand when the representational spaces in different elements of the semantic system converge with representational spaces in the evolving MEG signal. Participants performed a semantic-typicality judgment of written words drawn from nine different semantic categories in separate fMRI and MEG sessions. Results indicate an initial period of congruence between MEG and fMRI informational spaces dominated by the posterior inferior temporal gyrus and the ventral temporal cortex between 350-450 msec. This is followed by a second period of convergence between 450-795 msec where MEG and fMRI representational spaces conform in left angular gyrus and precuneus in addition to ventral temporal cortex. Results are consistent with the multistage recruitment of the semantic system, initially involving automatic aspects of the representational system and later extending to broader elements of the semantic system more strongly associated with internalised cognition.

Norms for pictures of proper names: contrasting famous people and well-known places in younger and older adults

Proper names comprise a class of labels that arbitrarily nominate specific entities, such as people and places. Compared to common nouns, retrieving proper names is more challenging. Thus, they constitute good alternative semantic categories for psycholinguistic and neurocognitive research and intervention. The ability to retrieve proper names is known to decrease with aging. Likewise, their retrieval may differ across their different categories (e.g., people and places) given their specific associated knowledge. Therefore, proper names' stimuli require careful selection due to their high dependence on prior experiences. Notably, normative datasets for pictures of proper names are scarce and hardly have considered the influence of aging and categories. The current study established culturally adapted norms for proper names' pictures (N = 80) from an adult sample (N = 107), in psycholinguistic measures (naming and categorization scores) and evaluative dimensions (fame, familiarity, distinctiveness, arousal, and representational quality). These norms were contrasted across different categories (famous people and well-known places) and age groups (younger and older adults). Additionally, the correlations between all variables were examined. Proper names' pictures were named and categorized above chance and overall rated as familiar, famous, distinctive, and of high representational quality. Age effects were observed across all variables, except familiarity. Category effects were occasionally observed. Finally, the correlations between the psycholinguistic measures and all rated dimensions suggest the relevance of controlling for these dimensions when assessing naming abilities. The current norms provide a relevant aging-adapted dataset that is publicly available for research and intervention purposes.

Deep neural networks reveal topic-level representations of sentences in medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus

When reading a sentence, individual words can be combined to create more complex meaning. In this study, we sought to uncover brain regions that reflect the representation of the meaning of sentences at the topic level, as opposed to the meaning of their individual constituent words when considered irrespective of their context. Using fMRI, we recorded the neural activity of participants while reading sentences. We constructed a topic-level sentence representations using the final layer of a convolutional neural network (CNN) trained to classify Wikipedia sentences into broad semantic categories. This model was contrasted with word-level sentence representations constructed using the average of the word embeddings constituting the sentence. Using representational similarity analysis, we found that the medial prefrontal cortex, lateral anterior temporal lobe, precuneus, and angular gyrus more strongly represent sentence topic-level, compared to word-level, meaning, uncovering the important role of these semantic system regions in the representation of topic-level meaning. Results were comparable when sentence meaning was modelled with a multilayer perceptron that was not sensitive to word order within a sentence, suggesting that the learning objective, in the terms of the topic being modelled, is the critical factor in capturing these neural representational spaces.

Exploring the Representations of Individual Entities in the Brain Combining EEG and Distributional Semantics

Semantic knowledge about individual entities (i.e., the referents of proper names such as Jacinta Ardern) is fine-grained, episodic, and strongly social in nature, when compared with knowledge about generic entities (the referents of common nouns such as politician). We investigate the semantic representations of individual entities in the brain; and for the first time we approach this question using both neural data, in the form of newly-acquired EEG data, and distributional models of word meaning, employing them to isolate semantic information regarding individual entities in the brain. We ran two sets of analyses. The first set of analyses is only concerned with the evoked responses to individual entities and their categories. We find that it is possible to classify them according to both their coarse and their fine-grained category at appropriate timepoints, but that it is hard to map representational information learned from individuals to their categories. In the second set of analyses, we learn to decode from evoked responses to distributional word vectors. These results indicate that such a mapping can be learnt successfully: this counts not only as a demonstration that representations of individuals can be discriminated in EEG responses, but also as a first brain-based validation of distributional semantic models as representations of individual entities. Finally, in-depth analyses of the decoder performance provide additional evidence that the referents of proper names and categories have little in common when it comes to their representation in the brain.

Proper names and personal identity

The present chapter reviews the body of knowledge acquired so far about the role of the temporal lobe in representing and processing proper names and individual identity information. This body of knowledge has been collected with the contribution of several methodologies, including neuroimaging, electrophysiological techniques, and, critically, clinical observations. All this evidence converges in showing that proper names and related information are processed in at least partially independent neural networks mainly placed in the anterior areas of the left temporal lobe. A description of the properties distinguishing proper names from common names is provided. These properties, it will be claimed, made a different anatomical organization necessary and, possibly, determined the evolution of the brain to support this advantageous distinction in meeting environmental demands.

Spatially Adjacent Regions in Posterior Cingulate Cortex Represent Familiar Faces at Different Levels of Complexity

Extensive research has shown that perceptual information of faces is processed in a network of hierarchically-organized areas within ventral temporal cortex. For familiar and famous faces, perceptual processing of faces is normally accompanied by extraction of semantic knowledge about the social status of persons. Semantic processing of familiar faces could entail progressive stages of information abstraction. However, the cortical mechanisms supporting multistage processing of familiar faces have not been characterized. Here, using an event-related fMRI experiment, familiar faces from four celebrity groups (actors, singers, politicians, and football players) and unfamiliar faces were presented to the human subjects (both males and females) while they were engaged in a face categorization task. We systematically explored the cortical representations for faces, familiar faces, subcategories of familiar faces, and familiar face identities using whole-brain univariate analysis, searchlight-based multivariate pattern analysis (MVPA), and functional connectivity analysis. Convergent evidence from all these analyses revealed a set of overlapping regions within posterior cingulate cortex (PCC) that contained decodable fMRI responses for representing different levels of semantic knowledge about familiar faces. Our results suggest a multistage pathway in PCC for processing semantic information of faces, analogous to the multistage pathway in ventral temporal cortex for processing perceptual information of faces.SIGNIFICANCE STATEMENT Recognizing familiar faces is an important component of social communications. Previous research has shown that a distributed network of brain areas is involved in processing the semantic information of familiar faces. However, it is not clear how different levels of semantic information are represented in the brain. Here, we evaluated the multivariate response patterns across the entire cortex to discover the areas that contain information for familiar faces, subcategories of familiar faces, and identities of familiar faces. The searchlight maps revealed that different levels of semantic information are represented in topographically adjacent areas within posterior cingulate cortex (PCC). The results suggest that semantic processing of faces is mediated through progressive stages of information abstraction in PCC.

fMRI response to automatic and purposeful familiar-face processing in perceptual and nonperceptual cortical regions

Viewing the faces of familiar people selectively activates a distributed network of brain regions implicated in both the perceptual and nonperceptual processing of conspecifics. In this functional magnetic resonance imaging (fMRI) study, we investigate the influence of depth of famous-face processing on this network, comparing a passive incidental face processing to a task that required the extraction of identity and biographic information. We observed that the precuneus, ventromedial prefrontal cortex (vmPFC), anterior temporal face patch (ATFP), and the amygdala exhibit a selective response even during incidental face processing. At the same time, face selectivity was enhanced in the lateral anterior temporal lobe (latATL) and the posterior superior temporal sulcus (pSTS) when identity and information extraction was required. In addition, goal-directed identity and information extraction was associated with a recruitment of inferior frontal gyrus (IFG), whereas this region was deactivated during passive viewing. Collectively, these results show that: 1) in addition to active information extraction, the extended system is recruited by the passive retrieval of person-related knowledge and 2) active access to such knowledge modulates activity in latATL and pSTS, potentially mediated via control circuits in the IFG.NEW & NOTEWORTHY Information is extracted from familiar faces in both automatic and active modes. Using functional MRI, we show: 1) that automatic access results in the selective activation of nonperceptual brain regions, the precuneus, ventromedial prefrontal cortex, and the anterior face patch and amygdala, demonstrating the automaticity of access to information in these regions; 2) selective increases in the activation of the lateral anterior temporal lobe and posterior superior temporal gyrus when biographic information is actively extracted.

fMRI-Indexed neural temporal tuning reveals the hierarchical organsiation of the face and person selective network

Recognising and knowing about conspecifics is vital to human interaction and is served in the brain by a well-characterised cortical network. Understanding the temporal dynamics of this network is critical to gaining insight into both hierarchical organisation and regional coordination. Here, we combine the high spatial resolution of fMRI with a paradigm that permits investigation of differential temporal tuning across cortical regions. We cognitively under- and overload the system using the rapid presentation (100-1200msec) of famous faces and buildings. We observed an increase in activity as presentation rates slowed and a negative deflection when inter-stimulus intervals (ISIs) were extended to longer periods. The primary distinction in tuning patterns was between core (perceptual) and extended (non-perceptual) systems but there was also evidence for nested hierarchies within systems, as well as indications of widespread parallel processing. Extended regions demonstrated common temporal tuning across regions which may indicate coordinated activity as they cooperate to manifest the diverse cognitive representation accomplished by this network. With the support of an additional psychophysical study, we demonstrated that ISIs necessary for different levels of semantic access are consistent with temporal tuning patterns. Collectively, these results show that regions of the person-knowledge network operate over different temporal timescales consistent with hierarchical organisation.

Intrinsic connectivity of anterior temporal lobe relates to individual differences in semantic retrieval for landmarks

Contemporary neuroscientific accounts suggest that ventral anterior temporal lobe (ATL) acts as a bilateral heteromodal semantic hub, which is particularly critical for the specific-level knowledge needed to recognise unique entities, such as familiar landmarks and faces. There may also be graded functional differences between left and right ATL, relating to effects of modality (linguistic versus non-linguistic) and category (e.g., knowledge of people and places). Individual differences in intrinsic connectivity from left and right ATL might be associated with variation in semantic categorisation performance across these categories and modalities. We recorded resting-state fMRI in 74 individuals and, in a separate session, examined semantic categorisation. People with greater connectivity between left and right ATL were more efficient at categorising landmarks (e.g., Eiffel Tower), especially when these were presented visually. In addition, participants who showed stronger connectivity from right than left ATL to medial occipital cortex showed more efficient semantic categorisation of landmarks regardless of modality of presentation. These results can be interpreted in terms of graded differences in the patterns of connectivity across left and right ATL, which give rise to a bilateral yet partially segregated semantic 'hub'. More specifically, right ATL connectivity supports the efficient semantic categorisation of landmarks.

Category Selectivity for Face and Scene Recognition in Human Medial Parietal Cortex

The rapid recognition and memory of faces and scenes implies the engagement of category-specific computational hubs in the ventral visual stream with the distributed cortical memory network. To better understand how recognition and identification occur in humans, we performed direct intracranial recordings, in a large cohort of patients (n = 50), from the medial parietal cortex (MPC) and the medial temporal lobe (MTL), structures known to be engaged during face and scene identification. We discovered that the MPC is topologically tuned to face and scene recognition, with clusters in MPC performing scene recognition bilaterally and face recognition in right subparietal sulcus. The MTL displayed a selectivity gradient with anterior, entorhinal cortex showing face selectivity and posterior parahippocampal regions showing scene selectivity. In both MPC and MTL, stimulus-specific identifiable exemplars led to greater activity in these cortical patches. These two regions work in concert for recognition of faces and scenes. Feature selectivity and identity-sensitive activity in the two regions was coincident, and they exhibited theta-phase locking during face and scene recognition. These findings together provide clear evidence for a specific role of subregions in the MPC for the recognition of unique entities.

General and feature-based semantic representations in the semantic network

How semantic representations are manifest over the brain remains a topic of active debate. A semantic representation may be determined by specific semantic features (e.g. sensorimotor information), or may abstract away from specific features and represent generalized semantic characteristics (general semantic representation). Here we tested whether nodes of the semantic system code for a general semantic representation and/or possess representational spaces linked to particular semantic features. In an fMRI study, eighteen participants performed a typicality judgment task with written words drawn from sixteen different categories. Multivariate pattern analysis (MVPA) and representational similarity analysis (RSA) were adopted to investigate the sensitivity of the brain regions to semantic content and the type of semantic representation coded (general or feature-based). We replicated previous findings of sensitivity to general semantic similarity in posterior middle/inferior temporal gyrus (pMTG/ITG) and precuneus (PC) and additionally observed general semantic representations in ventromedial prefrontal cortex (PFC). Finally, two brain regions of the semantic network were sensitive to semantic features: the left pMTG/ITG was sensitive to haptic perception and the left ventral temporal cortex (VTC) to size. This finding supports the involvement of both general semantic representation and feature-based representations in the brain's semantic system.

Regional Specialization and Coordination Within the Network for Perceiving and Knowing About Others

Seeing familiar faces prompts the recall of diverse kinds of person-related knowledge. How this information is encoded within the well-characterized face-/person-selective network remains an outstanding question. In this functional magnetic resonance imaging study, participants rated famous faces in 10 tasks covering 5 domains of person knowledge (social, episodic, semantic, physical, and nominal). Comparing different cognitive domains enabled us to 1) test the relative roles of brain regions in specific cognitive processes and 2) apply a multivariate network-level representational similarity analysis (NetRSA) to gain insight into underlying system-level organization. Comparing across cognitive domains revealed the importance of multiple domains in most regions, the importance of social over nominal knowledge in the anterior temporal lobe, and the functional subdivision of the temporoparietal junction into perceptual superior temporal sulcus and knowledge-related angular gyrus. NetRSA revealed a strong divide between regions implicated in "default-mode" cognition and the fronto-lateral elements that coordinated more with "core" perceptual components (fusiform/occipital face areas and posterior superior temporal sulcus). NetRSA also revealed a taxonomy of cognitive processes, with semantic retrieval being more similar to episodic than nominal knowledge. Collectively, these results illustrate the importance of coordinated activity of the person knowledge network in the instantiation of the diverse cognitive capacities of this system.

Cross Recruitment of Domain-Selective Cortical Representations Enables Flexible Semantic Knowledge

Knowledge about objects encompasses not only their prototypical features but also complex, atypical, semantic knowledge (e.g., "Pizza was invented in Naples"). This fMRI study of male and female human participants combines univariate and multivariate analyses to consider the cortical representation of this more complex semantic knowledge. Using the categories of food, people, and places, this study investigates whether access to spatially related geographic semantic knowledge (1) involves the same domain-selective neural representations involved in access to prototypical taste knowledge about food; and (2) elicits activation of neural representations classically linked to places when this geographic knowledge is accessed about food and people. In three experiments using word stimuli, domain-relevant and atypical conceptual access for the categories food, people, and places were assessed. Results uncover two principles of semantic representation: food-selective representations in the left insula continue to be recruited when prototypical taste knowledge is task-irrelevant and under conditions of high cognitive demand; access to geographic knowledge for food and people categories involves the additional recruitment of classically place-selective parahippocampal gyrus, retrosplenial complex, and transverse occipital sulcus. These findings underscore the importance of object category in the representation of a broad range of knowledge, while showing how the cross recruitment of specialized representations may endow the considerable flexibility of our complex semantic knowledge.SIGNIFICANCE STATEMENT We know not only stereotypical things about objects (an apple is round, graspable, edible) but can also flexibly combine typical and atypical features to form complex concepts (the metaphorical role an apple plays in Judeo-Christian belief). In this fMRI study, we observe that, when atypical geographic knowledge is accessed about food dishes, domain-selective sensorimotor-related cortical representations continue to be recruited, but that regions classically associated with place perception are additionally engaged. This interplay between categorically driven representations, linked to the object being accessed, and the flexible recruitment of semantic stores linked to the content being accessed, provides a potential mechanism for the broad representational repertoire of our semantic system.

Brain stimulation to left prefrontal cortex modulates attentional orienting to gaze cues

In social interactions, we rely on non-verbal cues like gaze direction to understand the behaviour of others. How we react to these cues is determined by the degree to which we believe that they originate from an entity with a mind capable of having internal states and showing intentional behaviour, a process called mind perception. While prior work has established a set of neural regions linked to mind perception, research has just begun to examine how mind perception affects social-cognitive mechanisms like gaze processing on a neuronal level. In the current experiment, participants performed a social attention task (i.e. attentional orienting to gaze cues) with either a human or a robot agent (i.e. manipulation of mind perception) while transcranial direct current stimulation (tDCS) was applied to prefrontal and temporo-parietal brain areas. The results show that temporo-parietal stimulation did not modulate mechanisms of social attention, neither in response to the human nor in response to the robot agent, whereas prefrontal stimulation enhanced attentional orienting in response to human gaze cues and attenuated attentional orienting in response to robot gaze cues. The findings suggest that mind perception modulates low-level mechanisms of social cognition via prefrontal structures, and that a certain degree of mind perception is essential in order for prefrontal stimulation to affect mechanisms of social attention. This article is part of the theme issue 'From social brains to social robots: applying neurocognitive insights to human-robot interaction'.

Seeing minds in others: Mind perception modulates low-level social-cognitive performance and relates to ventromedial prefrontal structures

In social interactions, we rely on nonverbal cues like gaze direction to understand the behavior of others. How we react to these cues is affected by whether they are believed to originate from an entity with a mind, capable of having internal states (i.e., mind perception). While prior work has established a set of neural regions linked to social-cognitive processes like mind perception, the degree to which activation within this network relates to performance in subsequent social-cognitive tasks remains unclear. In the current study, participants performed a mind perception task (i.e., judging the likelihood that faces, varying in physical human-likeness, have internal states) while event-related fMRI was collected. Afterwards, participants performed a social attention task outside the scanner, during which they were cued by the gaze of the same faces that they previously judged within the mind perception task. Parametric analyses of the fMRI data revealed that activity within ventromedial prefrontal cortex (vmPFC) was related to both mind ratings inside the scanner and gaze-cueing performance outside the scanner. In addition, other social brain regions were related to gaze-cueing performance, including frontal areas like the left insula, dorsolateral prefrontal cortex, and inferior frontal gyrus, as well as temporal areas like the left temporo-parietal junction and bilateral temporal gyri. The findings suggest that functions subserved by the vmPFC are relevant to both mind perception and social attention, implicating a role of vmPFC in the top-down modulation of low-level social-cognitive processes.

Concrete versus abstract forms of social concept: an fMRI comparison of knowledge about people versus social terms

The anterior temporal lobes (ATLs) play a key role in conceptual knowledge representation. The hub-and-spoke theory suggests that the contribution of the ATLs to semantic representation is (a) transmodal, i.e. integrating information from multiple sensorimotor and verbal modalities, and (b) pan-categorical, representing concepts from all categories. Another literature, however, suggests that this region's responses are modality- and category-selective; prominent examples include category selectivity for socially relevant concepts and face recognition. The predictions of each approach have never been directly compared. We used data from three studies to compare category-selective responses within the ATLs. Study 1 compared ATL responses to famous people versus another conceptual category (landmarks) from visual versus auditory inputs. Study 2 compared ATL responses to famous people from pictorial and written word inputs. Study 3 compared ATL responses to a different kind of socially relevant stimuli, namely abstract non-person-related words, in order to ascertain whether ATL subregions are engaged for social concepts more generally or only for person-related knowledge. Across all three studies a dominant bilateral ventral ATL cluster responded to all categories in all modalities. Anterior to this ‘pan-category’ transmodal region, a second cluster responded more weakly overall yet selectively for people, but did so equally for spoken names and faces (Study 1). A third region in the anterior superior temporal gyrus responded selectively to abstract socially relevant words (Study 3), but did not respond to concrete socially relevant words (i.e. written names; Study 2). These findings can be accommodated by the graded hub-and-spoke model of concept representation. On this view, the ventral ATL is the centre point of a bilateral ATL hub, which contributes to conceptual representation through transmodal distillation of information arising from multiple modality-specific association cortices. Partial specialization occurs across the graded ATL hub as a consequence of gradedly differential connectivity across the region.

This article is part of the theme issue ‘Varieties of abstract concepts: development, use and representation in the brain’.

The Artery of Aphasia, A Uniquely Sensitive Posterior Temporal Middle Cerebral Artery Branch that Supplies Language Areas in the Brain: Anatomy and Report of Four Cases

BACKGROUND

Arterial disruption during brain surgery can cause devastating injuries to wide expanses of white and gray matter beyond the tumor resection cavity. Such damage may occur as a result of disrupting blood flow through en passage arteries. Identification of these arteries is critical to prevent unforeseen neurologic sequelae during brain tumor resection. In this study, we discuss one such artery, termed the artery of aphasia (AoA), which when disrupted can lead to receptive and expressive language deficits.

METHODS

We performed a retrospective review of all patients undergoing an awake craniotomy for resection of a glioma by the senior author from 2012 to 2018. Patients were included if they experienced language deficits secondary to postoperative infarction in the left posterior temporal lobe in the distribution of the AoA. The gross anatomy of the AoA was then compared with activation likelihood estimations of the auditory and semantic language networks using coordinate-based meta-analytic techniques.

RESULTS

We identified 4 patients with left-sided posterior temporal artery infarctions in the distribution of the AoA on diffusion-weighted magnetic resonance imaging. All 4 patients developed substantial expressive and receptive language deficits after surgery. Functional language improvement occurred in only 2/4 patients. Activation likelihood estimations localized parts of the auditory and semantic language networks in the distribution of the AoA.

CONCLUSIONS

The AoA is prone to blood flow disruption despite benign manipulation. Patients seem to have limited capacity for speech recovery after intraoperative ischemia in the distribution of this artery, which supplies parts of the auditory and semantic language networks.

Temporal dynamics of access to amodal representations of category-level conceptual information

Categories describe semantic divisions between classes of objects and category-based models are widely used for investigation of the conceptual system. One critical issue in this endeavour is the isolation of conceptual from perceptual contributions to category-differences. An unambiguous way to address this confound is combining multiple input-modalities. To this end, we showed participants person/place stimuli using name and picture modalities. Using multivariate methods, we searched for category-sensitive neural patterns shared across input-modalities and thus independent from perceptual properties. The millisecond temporal resolution of magnetoencephalography (MEG) allowed us to consider the precise timing of conceptual access and, by confronting latencies between the two modalities ("time generalization"), how latencies of processing depends on the input-modality. Our results identified category-sensitive conceptual representations common between modalities at three stages and that conceptual access for words was delayed by about 90 msec with respect to pictures. We also show that for pictures, the first conceptual pattern of activity (shared between both words and pictures) occurs as early as 110 msec. Collectively, our results indicated that conceptual access at the category-level is a multistage process and that different delays in access across these two input-modalities determine when these representations are activated.

Category specificity in the medial temporal lobe: A systematic review

Theoretical accounts of medial temporal lobe (MTL) function ascribe different functions to subregions of the MTL including perirhinal, entorhinal, parahippocampal cortices, and the hippocampus. Some have suggested that the functional roles of these subregions vary in terms of their category specificity, showing preferential coding for certain stimulus types, but the evidence for this functional organization is mixed. In this systematic review, we evaluate existing evidence for regional specialization in the MTL for three categories of visual stimuli: faces, objects, and scenes. We review and synthesize across univariate and multivariate neuroimaging studies, as well as neuropsychological studies of cases with lesions to the MTL. Neuroimaging evidence suggests that faces activate the perirhinal cortex, entorhinal cortex, and the anterior hippocampus, while scenes engage the parahippocampal cortex and both the anterior and posterior hippocampus, depending on the contrast condition. There is some evidence for object-related activity in anterior MTL regions when compared to scenes, and in posterior MTL regions when compared to faces, suggesting that aspects of object representations may share similarities with face and scene representations. While neuroimaging evidence suggests some hippocampal specialization for faces and scenes, neuropsychological evidence shows that hippocampal damage leads to impairments in scene memory and perception, but does not entail equivalent impairments for faces in cases where the perirhinal cortex remains intact. Regional specialization based on stimulus categories has implications for understanding the mechanisms of MTL subregions, and highlights the need for the development of theoretical models of MTL function that can accommodate the differential patterns of specificity observed in the MTL.

Neural Representations of Belief Concepts: A Representational Similarity Approach to Social Semantics

The present experiment identified neural regions that represent a class of concepts that are independent of perceptual or sensory attributes. During functional magnetic resonance imaging scanning, participants viewed names of social groups (e.g. Atheists, Evangelicals, and Economists) and performed a one-back similarity judgment according to 1 of 2 dimensions of belief attributes: political orientation (Liberal to Conservative) or spiritualism (Spiritualist to Materialist). By generalizing across a wide variety of social groups that possess these beliefs, these attribute concepts did not coincide with any specific sensory quality, allowing us to target conceptual, rather than perceptual, representations. Multi-voxel pattern searchlight analysis was used to identify regions in which activation patterns distinguished the 2 ends of both dimensions: Conservative from Liberal social groups when participants focused on the political orientation dimension, and spiritual from Materialist groups when participants focused on the spiritualism dimension. A cluster in right precuneus exhibited such a pattern, indicating that it carries information about belief-attribute concepts and forms part of semantic memory—perhaps a component particularly concerned with psychological traits. This region did not overlap with the theory of mind network, which engaged nearby, but distinct, parts of precuneus. These findings have implications for the neural organization of conceptual knowledge, especially the understanding of social groups.

The neural representation of human versus nonhuman bipeds and quadrupeds

How do humans recognize humans among other creatures? Recent studies suggest that a preference for conspecifics may emerge already in perceptual processing, in regions such as the right posterior superior temporal sulcus (pSTS), implicated in visual perception of biological motion. In the current functional MRI study, participants viewed point-light displays of human and nonhuman creatures moving in their typical bipedal (man and chicken) or quadrupedal mode (crawling-baby and cat). Stronger activity for man and chicken versus baby and cat was found in the right pSTS responsive to biological motion. The novel effect of pedalism suggests that, if right pSTS contributes to recognizing of conspecifics, it does so by detecting perceptual features (e.g. bipedal motion) that reliably correlate with their appearance. A searchlight multivariate pattern analysis could decode humans and nonhumans across pedalism in the left pSTS and bilateral posterior cingulate cortex. This result implies a categorical human-nonhuman distinction, independent from within-category physical/perceptual variation. Thus, recognizing conspecifics involves visual classification based on perceptual features that most frequently co-occur with humans, such as bipedalism, and retrieval of information that determines category membership above and beyond visual appearance. The current findings show that these processes are at work in separate brain networks.

Plastic reorganization of neural systems for perception of others in the congenitally blind

Recent evidence suggests that the function of the core system for face perception might extend beyond visual face-perception to a broader role in person perception. To critically test the broader role of core face-system in person perception, we examined the role of the core system during the perception of others in 7 congenitally blind individuals and 15 sighted subjects by measuring their neural responses using fMRI while they listened to voices and performed identity and emotion recognition tasks. We hypothesised that in people who have had no visual experience of faces, core face-system areas may assume a role in the perception of others via voices. Results showed that emotions conveyed by voices can be decoded in homologues of the core face system only in the blind. Moreover, there was a specific enhancement of response to verbal as compared to non-verbal stimuli in bilateral fusiform face areas and the right posterior superior temporal sulcus showing that the core system also assumes some language-related functions in the blind. These results indicate that, in individuals with no history of visual experience, areas of the core system for face perception may assume a role in aspects of voice perception that are relevant to social cognition and perception of others' emotions.

Two Distinct Scene-Processing Networks Connecting Vision and Memory

A number of regions in the human brain are known to be involved in processing natural scenes, but the field has lacked a unifying framework for understanding how these different regions are organized and interact. We provide evidence from functional connectivity and meta-analyses for a new organizational principle, in which scene processing relies upon two distinct networks that split the classically defined parahippocampal place area (PPA). The first network of strongly connected regions consists of the occipital place area/transverse occipital sulcus and posterior PPA, which contain retinotopic maps and are not strongly coupled to the hippocampus at rest. The second network consists of the caudal inferior parietal lobule, retrosplenial complex, and anterior PPA, which connect to the hippocampus (especially anterior hippocampus), and are implicated in both visual and nonvisual tasks, including episodic memory and navigation. We propose that these two distinct networks capture the primary functional division among scene-processing regions, between those that process visual features from the current view of a scene and those that connect information from a current scene view with a much broader temporal and spatial context. This new framework for understanding the neural substrates of scene-processing bridges results from many lines of research, and makes specific functional predictions.

Object Domain and Modality in the Ventral Visual Pathway

The nature of domain-specific organization in higher-order visual cortex (ventral occipital temporal cortex, VOTC) has been investigated both in the case of visual experience deprivation and of modality of stimulation in sighted individuals. Object domain interacts in an intriguing and revelatory way with visual experience and modality of stimulation: selectivity for artifacts and scene domains is largely immune to visual deprivation and is multi-modal, whereas selectivity for animate items in lateral posterior fusiform gyrus is present only with visual stimulation. This domain-by-modality interaction is not readily accommodated by existing theories of VOTC representation. We conjecture that these effects reflect a distinction between the visual features that characterize different object domains and their interaction with different types of downstream computational systems.

Outside Looking In: Landmark Generalization in the Human Navigational System

The use of landmarks is central to many navigational strategies. Here we use multivoxel pattern analysis of fMRI data to understand how landmarks are coded in the human brain. Subjects were scanned while viewing the interiors and exteriors of campus buildings. Despite their visual dissimilarity, interiors and exteriors corresponding to the same building elicited similar activity patterns in the parahippocampal place area (PPA), retrosplenial complex (RSC), and occipital place area (OPA), three regions known to respond strongly to scenes and buildings. Generalization across stimuli depended on knowing the correspondences among them in the PPA but not in the other two regions, suggesting that the PPA is the key region involved in learning the different perceptual instantiations of a landmark. In contrast, generalization depended on the ability to freely retrieve information from memory in RSC, and it did not depend on familiarity or cognitive task in OPA. Together, these results suggest a tripartite division of labor, whereby PPA codes landmark identity, RSC retrieves spatial or conceptual information associated with landmarks, and OPA processes visual features that are important for landmark recognition.

SIGNIFICANCE STATEMENT

A central element of spatial navigation is the ability to recognize the landmarks that mark different places in the world. However, little is known about how the brain performs this function. Here we show that the parahippocampal place area (PPA), a region in human occipitotemporal cortex, exhibits key features of a landmark recognition mechanism. Specifically, the PPA treats different perceptual instantiations of the same landmark as representationally similar, but only when subjects have enough experience to know the correspondences among the stimuli. We also identify two other brain regions that exhibit landmark generalization, but with less sensitivity to familiarity. These results elucidate the brain networks involved in the learning and recognition of navigational landmarks.

In our own image? Emotional and neural processing differences when observing human-human vs human-robot interactions

Notwithstanding the significant role that human-robot interactions (HRI) will play in the near future, limited research has explored the neural correlates of feeling eerie in response to social robots. To address this empirical lacuna, the current investigation examined brain activity using functional magnetic resonance imaging while a group of participants (n = 26) viewed a series of human-human interactions (HHI) and HRI. Although brain sites constituting the mentalizing network were found to respond to both types of interactions, systematic neural variation across sites signaled diverging social-cognitive strategies during HHI and HRI processing. Specifically, HHI elicited increased activity in the left temporal-parietal junction indicative of situation-specific mental state attributions, whereas HRI recruited the precuneus and the ventromedial prefrontal cortex (VMPFC) suggestive of script-based social reasoning. Activity in the VMPFC also tracked feelings of eeriness towards HRI in a parametric manner, revealing a potential neural correlate for a phenomenon known as the uncanny valley. By demonstrating how understanding social interactions depends on the kind of agents involved, this study highlights pivotal sub-routes of impression formation and identifies prominent challenges in the use of humanoid robots.

From Parts to Identity: Invariance and Sensitivity of Face Representations to Different Face Halves

Recognizing the identity of a face is computationally challenging, because it requires distinguishing between similar images depicting different people, while recognizing even very different images depicting a same person. Previous human fMRI studies investigated representations of face identity in the presence of changes in viewpoint and in expression. Despite the importance of holistic processing for face recognition, an investigation of representations of face identity across different face parts is missing. To fill this gap, we investigated representations of face identity and their invariance across different face halves. Information about face identity with invariance across changes in the face half was individuated in the right anterior temporal lobe, indicating this region as the most plausible candidate brain area for the representation of face identity. In a complementary analysis, information distinguishing between different face halves was found to decline along the posterior to anterior axis in the ventral stream.

Nonmotor aspects of action concepts

Reading an action verb elicits the retrieval of its associated body movements as well as its typical goal-the outcome to which it is directed. Two fMRI experiments are reported in which retrieval of goal attributes was isolated from retrieval of motoric ones by contrasting actions that are either done intentionally (e.g., drink) and thus have associated goal information or by accident (e.g., hiccup). Orthogonally, the actions also varied in their motoricity (e.g., drink vs. imagine). Across both levels of motoricity, goal-directedness influenced the activity of a portion of left posterior inferior parietal lobe (pIPL). These effects were not explicable by the grammatical properties, imageability, or amount of body movement associated with these different types of verbs. In contrast, motoricity (across levels of goal-directedness) activated primarily the left middle temporal gyrus. Furthermore, pIPL was found to be distinct from the portion of left parietal lobe implicated in theory of mind, as localized in the same participants. This is consistent with the observation that pIPL contains many functionally distinct subregions and that some of these support conceptual knowledge. The present findings illustrate that, in particular, the pIPL is involved in representing attributes of intentional actions, likely their typical goals, but not their associated body movements. This result serves to describe an attribute-selective semantic subsystem for at least one type of nonmotor aspect of action knowledge.

Brain regions that represent amodal conceptual knowledge

To what extent do the brain regions implicated in semantic processing contribute to the representation of amodal conceptual content rather than modality-specific mechanisms or mechanisms of semantic access and manipulation? Here, we propose that a brain region can be considered to represent amodal conceptual object knowledge if it is supramodal and plays a role in distinguishing among the conceptual representations of different objects. In an fMRI study, human participants made category typicality judgments about pictured objects or their names drawn from five different categories. Crossmodal multivariate pattern analysis revealed a network of six left-lateralized regions largely outside of category-selective visual cortex that showed a supramodal representation of object categories. These were located in the posterior middle/inferior temporal gyrus (pMTG/ITG), angular gyrus, ventral temporal cortex, posterior cingulate/precuneus (PC), and lateral and dorsomedial prefrontal cortex. Representational similarity analysis within these regions determined that the similarity between category-specific patterns of neural activity in the pMTG/ITG and the PC was consistent with the semantic similarity between these categories. This finding supports the PC and pMTG/ITG as candidate regions for the amodal representation of the conceptual properties of objects.