Neural foundations for understanding social and mechanical concepts

A quantitative comparison of atlas parcellations on the human superior temporal sulcus

The superior temporal sulcus (STS) has a functional topography that has been difficult to characterize through traditional approaches. Automated atlas parcellations may be one solution while also being beneficial for both dimensional reduction and standardizing regions of interest, but they yield very different boundary definitions along the STS. Here we evaluate how well machine learning classifiers can correctly identify six social cognitive tasks from STS activation patterns dimensionally reduced using four popular atlases (Glasser et al., 2016; Gordon et al., 2016; Power et al., 2011 as projected onto the surface by Arslan et al., 2018; Schaefer et al., 2018). Functional data was summarized within each STS parcel in one of four ways, then subjected to leave-one-subject-out cross-validation SVM classification. We found that the classifiers could readily label conditions when data was parcellated using any of the four atlases, evidence that dimensional reduction to parcels did not compromise functional fingerprints. Mean activation for the social conditions was the most effective metric for classification in the right STS, whereas all the metrics classified equally well in the left STS. Interestingly, even atlases constructed from random parcellation schemes (null atlases) classified the conditions with high accuracy. We therefore conclude that the complex activation maps on the STS are readily differentiated at a coarse granular level, despite a strict topography having not yet been identified. Further work is required to identify what features have greatest potential to improve the utility of atlases in replacing functional localizers.

Distributed representations of behaviour-derived object dimensions in the human visual system

Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioural goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of dimensions derived from a large-scale analysis of human similarity judgements directly onto the brain. Our results reveal broadly distributed representations of behaviourally relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behaviour-derived dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.

Distributed representations of behavior-derived object dimensions in the human visual system

Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioral goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of dimensions derived from a large-scale analysis of human similarity judgments directly onto the brain. Our results reveal broadly distributed representations of behaviorally-relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behavior-derived dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.

Two Sides of Theory of Mind: Mental State Attribution to Moving Shapes in Paranoid Schizophrenia Is Independent of the Severity of Positive Symptoms

BACKGROUND

Theory of Mind (ToM) impairment has repeatedly been found in paranoid schizophrenia. The current study aims at investigating whether this is related to a deficit in ToM (undermentalizing) or an increased ToM ability to hyperattribute others' mental states (overmentalizing).

METHODS

Mental state attribution was examined in 24 patients diagnosed with schizophrenia (12 acute paranoid (APS) and 12 post-acute paranoid (PPS)) with regard to positive symptoms as well as matched healthy persons using a moving shapes paradigm. We used 3-T-functional magnetic resonance imaging (fMRI) to provide insights into the neural underpinnings of ToM due to attributional processes in different states of paranoid schizophrenia.

RESULTS

In the condition that makes demands on theory of mind skills (ToM condition), in patients with diagnosed schizophrenia less appropriate mental state descriptions have been used, and they attributed mental states less often to the moving shapes than healthy persons. On a neural level, patients suffering from schizophrenia exhibited within the ToM network hypoactivity in the medial prefrontal cortex (MPFC) and hyperactivity in the temporo-parietal junction (TPJ) as compared to the healthy sample.

CONCLUSIONS

Our results indicate both undermentalizing and hypoactivity in the MPFC and increased overattribution related to hyperactivity in the TPJ in paranoid schizophrenia, providing new implications for understanding ToM in paranoid schizophrenia.

Overlapping representations of observed actions and action-related features

The lateral occipitotemporal cortex (LOTC) has been shown to capture the representational structure of a smaller range of actions. In the current study, we carried out an fMRI experiment in which we presented human participants with images depicting 100 different actions and used representational similarity analysis (RSA) to determine which brain regions capture the semantic action space established using judgments of action similarity. Moreover, to determine the contribution of a wide range of action-related features to the neural representation of the semantic action space we constructed an action feature model on the basis of ratings of 44 different features. We found that the semantic action space model and the action feature model are best captured by overlapping activation patterns in bilateral LOTC and ventral occipitotemporal cortex (VOTC). An RSA on eight dimensions resulting from principal component analysis carried out on the action feature model revealed partly overlapping representations within bilateral LOTC, VOTC, and the parietal lobe. Our results suggest spatially overlapping representations of the semantic action space of a wide range of actions and the corresponding action-related features. Together, our results add to our understanding of the kind of representations along the LOTC that support action understanding.

When shapes are more than shapes: perceptual, developmental, and neurophysiological basis for attributions of animacy and theory of mind

Among a variety of entities in their environment, what do humans consider alive or animate and how does this attribution of animacy promote development of more abstract levels of mentalizing? By decontextualizing the environment of bodily features, we review how physical movements give rise to perceived animacy in Heider-Simmel style animations. We discuss the developmental course of how perceived animacy shapes our interpretation of the social world, and specifically discuss when and how children transition from perceiving actions as goal-directed to attributing behaviors to unobservable mental states. This transition from a teleological stance, asserting a goal-oriented interpretation to an agent's actions, to a mentalistic stance allows older children to reason about more complex actions guided by hidden beliefs. The acquisition of these more complex cognitive behaviors happens developmentally at the same time neural systems for social cognition are coming online in young children. We review perceptual, developmental, and neural evidence to identify the joint cognitive and neural changes associated with when children begin to mentalize and how this ability is instantiated in the brain.

Understanding Human Object Vision: A Picture Is Worth a Thousand Representations

Objects are the core meaningful elements in our visual environment. Classic theories of object vision focus upon object recognition and are elegant and simple. Some of their proposals still stand, yet the simplicity is gone. Recent evolutions in behavioral paradigms, neuroscientific methods, and computational modeling have allowed vision scientists to uncover the complexity of the multidimensional representational space that underlies object vision. We review these findings and propose that the key to understanding this complexity is to relate object vision to the full repertoire of behavioral goals that underlie human behavior, running far beyond object recognition. There might be no such thing as core object recognition, and if it exists, then its importance is more limited than traditionally thought.

Cognitive load affects early processes involved in mentalizing robot behaviour

How individuals interpret robots' actions is a timely question in the context of the general approach to increase robot's presence in human social environment in the decades to come. Facing robots, people might have a tendency to explain their actions in mentalistic terms, granting them intentions. However, how default or controllable this process is still under debate. In four experiments, we asked participants to choose between mentalistic (intentional) and mechanistic (non-intentional) descriptions to describe depicted actions of a robot in various scenarios. Our results show the primacy of mentalistic descriptions that are processed faster than mechanistic ones (experiment 1). This effect was even stronger under high vs low cognitive load when people had to decide between the two alternatives (experiment 2). Interestingly, while there was no effect of cognitive load at the later stages of the processing arguing for controllability (experiment 3), imposing cognitive load on participants at an early stage of observation resulted in a faster attribution of mentalistic properties to the robot (experiment 4). We discuss these results in the context of the idea that social cognition is a default system.

Precuneus brain response changes differently during human-robot and human-human dyadic social interaction

Human–human interactions (HHI) and human–robot interactions (HRI) are compared to identify differences between cognitive processes reflecting bonding in social interactions with natural and artificial agents. We capitalize on a unique corpus of neuroimaging data (fMRI) recorded while participants freely discussed with another human or a conversational robotic head, in order to study a crucial parameter of human social cognition, namely that social interactions are adaptive bidirectional processes that evolve over time. We used linear statistics to identify regions of the brain where activity changes differently when participants carry out twelve one-minute conversations, alternating between a human and a robotic interlocutor. Results show that activity in the posterior cingulate cortex, a key region associated with social cognition, increases over time in HHI but not in HRI. These results are interpreted as reflecting a process of strengthening social bonding during repeated exchanges when the interacting agent is a human, but not a robot.

Unique contributions of perceptual and conceptual humanness to object representations in the human brain

The human brain is able to quickly and accurately identify objects in a dynamic visual world. Objects evoke different patterns of neural activity in the visual system, which reflect object category memberships. However, the underlying dimensions of object representations in the brain remain unclear. Recent research suggests that objects similarity to humans is one of the main dimensions used by the brain to organise objects, but the nature of the human-similarity features driving this organisation are still unknown. Here, we investigate the relative contributions of perceptual and conceptual features of humanness to the representational organisation of objects in the human visual system. We collected behavioural judgements of human-similarity of various objects, which were compared with time-resolved neuroimaging responses to the same objects. The behavioural judgement tasks targeted either perceptual or conceptual humanness features to determine their respective contribution to perceived human-similarity. Behavioural and neuroimaging data revealed significant and unique contributions of both perceptual and conceptual features of humanness, each explaining unique variance in neuroimaging data. Furthermore, our results showed distinct spatio-temporal dynamics in the processing of conceptual and perceptual humanness features, with later and more lateralised brain responses to conceptual features. This study highlights the critical importance of social requirements in information processing and organisation in the human brain.

Personal space regulation is affected by unilateral temporal lesions beyond the amygdala

We constantly face situations involving interactions with others that require us to automatically adjust our physical distances to avoid discomfort or anxiety. A previous case study has demonstrated that the integrity of both amygdalae is essential to regulate interpersonal distances. Despite unilateral lesion to the amygdala, as to other sectors of the medial temporal cortex, are known to also affect social behavior, their role in the regulation of interpersonal distances has never been investigated. Here, we sought to fill this gap by testing three patients with unilateral temporal lesions following surgical resections, including one patient with a lesion mainly centered on the amygdala and two with lesions to adjacent medial temporal cortex, on two versions of the stop distance paradigm (i.e. in a virtual reality environment and in a real setting). Our results showed that all three patients set shorter interpersonal distances compared to neurotypical controls. In addition, compared to controls, none of the patients adjusted such physical distances depending on facial emotional expressions, despite they preserved ability to categorize them. Finally, patients' heart rate responses differed from controls when viewing approaching faces. Our findings bring compelling evidence that unilateral lesions within the medial temporal cortex, not necessarily restricted to the amygdala, are sufficient to alter interpersonal distance, thus shedding new light on the neural circuitry regulating distance in social interactions.

Virtual Reality for Safe Testing and Development in Collaborative Robotics: Challenges and Perspectives

Collaborative robots (cobots) could help humans in tasks that are mundane, dangerous or where direct human contact carries risk. Yet, the collaboration between humans and robots is severely limited by the aspects of the safety and comfort of human operators. In this paper, we outline the use of extended reality (XR) as a way to test and develop collaboration with robots. We focus on virtual reality (VR) in simulating collaboration scenarios and the use of cobot digital twins. This is specifically useful in situations that are difficult or even impossible to safely test in real life, such as dangerous scenarios. We describe using XR simulations as a means to evaluate collaboration with robots without putting humans at harm. We show how an XR setting enables combining human behavioral data, subjective self-reports, and biosignals signifying human comfort, stress and cognitive load during collaboration. Several works demonstrate XR can be used to train human operators and provide them with augmented reality (AR) interfaces to enhance their performance with robots. We also provide a first attempt at what could become the basis for a human–robot collaboration testing framework, specifically for designing and testing factors affecting human–robot collaboration. The use of XR has the potential to change the way we design and test cobots, and train cobot operators, in a range of applications: from industry, through healthcare, to space operations.

Using High-Density Electroencephalography to Explore Spatiotemporal Representations of Object Categories in Visual Cortex

Visual object perception involves neural processes that unfold over time and recruit multiple regions of the brain. Here, we use high-density EEG to investigate the spatiotemporal representations of object categories across the dorsal and ventral pathways. In , human participants were presented with images from two animate object categories (birds and insects) and two inanimate categories (tools and graspable objects). In , participants viewed images of tools and graspable objects from a different stimulus set, one in which a shape confound that often exists between these categories (elongation) was controlled for. To explore the temporal dynamics of object representations, we employed time-resolved multivariate pattern analysis on the EEG time series data. This was performed at the electrode level as well as in source space of two regions of interest: one encompassing the ventral pathway and another encompassing the dorsal pathway. Our results demonstrate shape, exemplar, and category information can be decoded from the EEG signal. Multivariate pattern analysis within source space revealed that both dorsal and ventral pathways contain information pertaining to shape, inanimate object categories, and animate object categories. Of particular interest, we note striking similarities obtained in both ventral stream and dorsal stream regions of interest. These findings provide insight into the spatio-temporal dynamics of object representation and contribute to a growing literature that has begun to redefine the traditional role of the dorsal pathway.

Using game-like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

This study introduces a novel, game-like method for measuring social intelligence: the Social Shapes Test. Unlike other existing video or game-based tests, the Shapes Test uses animations of abstract shapes to represent social interactions. We explore demographic differences in Shapes Test scores compared to a written situational judgment test. Gender and race/ethnicity only had meaningful effects on written SJT scores while no effects were found for Shapes Test scores. This pattern of results remained after controlling for general mental ability and English language exposure. We also found metric invariance between demographic groups for both tests. Our results demonstrate the potential for using animated shape tasks as an alternative to written SJTs when designing future game-based assessments.

The role of animal faces in the animate-inanimate distinction in the ventral temporal cortex

Animate and inanimate objects elicit distinct response patterns in the human ventral temporal cortex (VTC), but the exact features driving this distinction are still poorly understood. One prominent feature that distinguishes typical animals from inanimate objects and that could potentially explain the animate-inanimate distinction in the VTC is the presence of a face. In the current fMRI study, we investigated this possibility by creating a stimulus set that included animals with faces, faceless animals, and inanimate objects, carefully matched in order to minimize other visual differences. We used both searchlight-based and ROI-based representational similarity analysis (RSA) to test whether the presence of a face explains the animate-inanimate distinction in the VTC. The searchlight analysis revealed that when animals with faces were removed from the analysis, the animate-inanimate distinction almost disappeared. The ROI-based RSA revealed a similar pattern of results, but also showed that, even in the absence of faces, information about agency (a combination of animal's ability to move and think) is present in parts of the VTC that are sensitive to animacy. Together, these analyses showed that animals with faces do elicit a stronger animate/inanimate response in the VTC, but that faces are not necessary in order to observe high-level animacy information (e.g., agency) in parts of the VTC. A possible explanation could be that this animacy-related activity is driven not by faces per se, or the visual features of faces, but by other factors that correlate with face presence, such as the capacity for self-movement and thought. In short, the VTC might treat the face as a proxy for agency, a ubiquitous feature of familiar animals.

Domain-specific connectivity drives the organization of object knowledge in the brain

The goal of this chapter is to review neuropsychological and functional MRI findings that inform a theory of the causes of functional specialization for semantic categories within occipito-temporal cortex-the ventral visual processing pathway. The occipito-temporal pathway supports visual object processing and recognition. The theoretical framework that drives this review considers visual object recognition through the lens of how "downstream" systems interact with the outputs of visual recognition processes. Those downstream processes include conceptual interpretation, grasping and object use, navigating and orienting in an environment, physical reasoning about the world, and inferring future actions and the inner mental states of agents. The core argument of this chapter is that innately constrained connectivity between occipito-temporal areas and other regions of the brain is the basis for the emergence of neural specificity for a limited number of semantic domains in the brain.

Two 'what' pathways for action and object recognition

The ventral visual stream is conceived as a pathway for object recognition. However, we also recognize the actions an object can be involved in. Here, we show that action recognition critically depends on a pathway in lateral occipitotemporal cortex, partially overlapping and topographically aligned with object representations that are precursors for action recognition. By contrast, object features that are more relevant for object recognition, such as color and texture, are typically found in ventral occipitotemporal cortex. We argue that occipitotemporal cortex contains similarly organized lateral and ventral 'what' pathways for action and object recognition, respectively. This account explains a number of observed phenomena, such as the duplication of object domains and the specific representational profiles in lateral and ventral cortex.

What tool representation, intuitive physics, and action have in common: The brain's first-person physics engine

An overlapping set of brain regions in parietal and frontal cortex are engaged by different types of tasks and stimuli: (i) making inferences about the physical structure and dynamics of the world, (ii) passively viewing, or actively interacting with, manipulable objects, and (iii) planning and execution of reaching and grasping actions. We suggest the observed neural overlap is because a common superordinate computation is engaged by each of those different tasks: A forward model of physical reasoning about how first-person actions will affect the world and be affected by unfolding physical events. This perspective offers an account of why some physical predictions are systematically incorrect - there can be a mismatch between how physical scenarios are experimentally framed and the native format of the inferences generated by the brain's first-person physics engine. This perspective generates new empirical expectations about the conditions under which physical reasoning may exhibit systematic biases.

Do shapes have feelings? Social attribution in children with autism spectrum disorder and attention-deficit/hyperactivity disorder

Theory of mind (ToM) deficits are common in children with neurodevelopmental disorders (NDDs), such as autism spectrum disorder (ASD) and attention-deficit/hyperactivity disorder (ADHD), which contribute to their social and cognitive difficulties. The social attribution task (SAT) involves geometrical shapes moving in patterns that depict social interactions and is known to recruit brain regions from the classic ToM network. To better understand ToM in ASD and ADHD children, we examined the neural correlates using the SAT and functional magnetic resonance imaging (fMRI) in a cohort of 200 children: ASD (N = 76), ADHD (N = 74) and typically developing (TD; N = 50) (4-19 years). In the scanner, participants were presented with SAT videos corresponding to social help, social threat, and random conditions. Contrasting social vs. random, the ASD compared with TD children showed atypical activation in ToM brain areas-the middle temporal and anterior cingulate gyri. In the social help vs. social threat condition, atypical activation of the bilateral middle cingulate and right supramarginal and superior temporal gyri was shared across the NDD children, with between-diagnosis differences only being observed in the right fusiform. Data-driven subgrouping identified two distinct subgroups spanning all groups that differed in both their clinical characteristics and brain-behaviour relations with ToM ability.

Cortical Activation to Social and Mechanical Stimuli in the Infant Brain

From the early days of life infants distinguish between social and non-social physical entities and have different expectations for the way these two entities should move and interact. At the same time, we know very little about the cortical systems that support this early emerging ability. The goal of the current research was to assess the extent to which infant's processing of social and non-social physical entities is mediated by distinct information processing systems in the temporal cortex. Using a cross-sectional design, infants aged 6-9 months (Experiment 1) and 11-18 months (Experiment 2) were presented with two types of events: social interaction and mechanical interaction. In the social interaction event (patterned after Hamlin et al., 2007), an entity with googly eyes, hair tufts, and an implied goal of moving up the hill was either helped up, or pushed down, a hill through the actions of another social entity. In the mechanical interaction event, the googly eyes and hair tufts were replaced with vertical black dots and a hook and clasp, and the objects moved up or down the hill via mechanical interactions. FNIRS was used to measure activation from temporal cortex while infants viewed the test events. In both age groups, viewing social and mechanical interaction events elicited different patterns of activation in the right temporal cortex, although responses were more specialized in the older age group. Activation was not obtained in these areas when the objects moved in synchrony without interacting, suggesting that the causal nature of the interaction events may be responsible, in part, to the results obtained. This is one of the few fNIRS studies that has investigated age-related patterns of cortical activation and the first to provide insight into the functional development of networks specialized for processing of social and non-social physical entities engaged in interaction events.

Attributing social meaning to animated shapes: A new experimental study of apparent behavior

In 1944, Heider and Simmel reported that observers could perceive simple animated geometric shapes as characters with emotions, intentions, and other social attributes. This work has been cited over 3000 times and has had wide and ongoing influence on the study of social cognition and social intelligence. However, many researchers in this area have continued to use the original Heider and Simmel black-and-white video. We asked whether the original findings could be reproduced 75 years later by creating 32 new colored animated shape videos designed to depict various social plots and testing whether they can evoke similar spontaneous social attributions. Participants (N = 66) viewed our videos and were asked to write narratives which we coded for indicia of different types of social attributions. Consistent with Heider and Simmel, we found that participants spontaneously attributed social meaning to the videos. We observed that responses to our videos were also similar to responses to the original video reported by Klin (2000), despite being only 13-23 s and portraying a broader range of social plots. Participants varied in how many social attributions they made in response, and the videos varied in how much they elicited such responses. Our set of animated shape videos is freely available online for all researchers to use and forms the basis of a multiple-choice assessment of social intelligence (Brown et al., 2019).

The preterm social brain: altered functional networks for Theory of Mind in very preterm children

Neurodevelopmental difficulties emerge in very preterm born children (<32-week gestation) in infancy and continue to early adulthood but little is known about their social-cognitive development. This study utilized the complementary methodological advantages of both functional MRI and magnetoencephalography to examine the neural underpinnings of Theory of Mind in very preterm birth. Theory of Mind, one of the core social-cognitive skills, is the ability to attribute mental states to others, and is crucial for predicting others’ behaviours in social interactions. Eighty-three children (40 very preterm born, 24 boys, age = 8.7 ± 0.5 years, and 43 full-term born, 22 boys, age = 8.6 ± 0.5 years) completed the study. In functional MRI, both groups recruited classic Theory of Mind areas, without significant group differences. However, reduced Theory of Mind connectivity in the very preterm born group was found in magnetoencephalography in distinct theta, alpha and beta-band networks anchored in a set of brain regions that comprise the social brain. These networks included regions such as the angular gyrus, the medial pre-frontal cortex, the superior temporal gyrus and the temporal poles. Very preterm born children showed increased connectivity compared to controls in a network anchored in the occipital gyri rather than classical social-processing regions. Very preterm born children made significantly more attribution errors and mis-construed the social scenarios. Findings offer novel insight into the neural networks, supporting social cognition in very preterm born children and highlight the importance of multimodal neuroimaging to interrogate the social brain in clinical populations.

Frontotemporal dementia, music perception and social cognition share neurobiological circuits: A meta-analysis

Frontotemporal dementia (FTD) is a neurodegenerative disease that presents with profound changes in social cognition. Music might be a sensitive probe for social cognition abilities, but underlying neurobiological substrates are unclear. We performed a meta-analysis of voxel-based morphometry studies in FTD patients and functional MRI studies for music perception and social cognition tasks in cognitively normal controls to identify robust patterns of atrophy (FTD) or activation (music perception or social cognition). Conjunction analyses were performed to identify overlapping brain regions. In total 303 articles were included: 53 for FTD (n = 1153 patients, 42.5% female; 1337 controls, 53.8% female), 28 for music perception (n = 540, 51.8% female) and 222 for social cognition in controls (n = 5664, 50.2% female). We observed considerable overlap in atrophy patterns associated with FTD, and functional activation associated with music perception and social cognition, mostly encompassing the ventral language network. We further observed overlap across all three modalities in mesolimbic, basal forebrain and striatal regions. The results of our meta-analysis suggest that music perception and social cognition share neurobiological circuits that are affected in FTD. This supports the idea that music might be a sensitive probe for social cognition abilities with implications for diagnosis and monitoring.

Dynamic Content Reactivation Supports Naturalistic Autobiographical Recall in Humans

Humans can vividly recall and re-experience events from their past, and these are commonly referred to as episodic or autobiographical memories. fMRI experiments reliably associate autobiographical event recall with activity in a network of "default" or "core" brain regions. However, as prior studies have relied on covert (silent) recall procedures, current understanding may be hampered by methodological limitations that obscure dynamic effects supporting moment-to-moment content retrieval. Here, fMRI participants (N = 40) overtly (verbally) recalled memories for ∼2 min periods. The content of spoken descriptions was categorized using a variant of the Autobiographical Interview (AI) procedure (Levine et al., 2002) and temporally re-aligned with BOLD data so activity accompanying the recall of different details could be measured. Replicating prior work, sustained effects associated with autobiographical recall periods (which are insensitive to the moment-to-moment content of retrieval) fell primarily within canonical default network regions. Spoken descriptions were rich in episodic details, frequently focusing on physical entities, their ongoing activities, and their appearances. Critically, neural activity associated with recalling specific details (e.g., those related to people or places) was transient, broadly distributed, and grounded in category-selective cortex (e.g., regions related to social cognition or scene processing). Thus, although a single network may generally support the process of vivid event reconstruction, the structures required to provide detail-related information shift in a predictable manner that respects domain-level representations across the cortex.SIGNIFICANCE STATEMENT Humans can vividly recall memories of autobiographical episodes, a process thought to involve the reconstruction of numerous distinct event details. Yet how the brain represents a complex episode as it unfolds over time remains unclear and appears inconsistent across experimental traditions. One hurdle is the use of covert (silent) in-scanner recall to study autobiographical memory, which prevents experimenter knowledge of what information is being retrieved, and when, throughout the remembering process. In this experiment, participants overtly described autobiographical memories while undergoing fMRI. Activity associated with the recall and description of specific details was transient, broadly distributed, and grounded in category-selective cortex. Thus, it appears that as events unfold mentally, structures are dynamically reactivated to support vivid recollection.

The Role of Amygdala in Self-Conscious Emotions in a Patient With Acquired Bilateral Damage

Shame plays a fundamental role in the regulation of our social behavior. One intriguing question is whether amygdala might play a role in processing this emotion. In the present single-case study, we tested a patient with acquired damage of bilateral amygdalae and surrounding areas as well as healthy controls on shame processing and other social cognitive tasks. Results revealed that the patient's subjective experience of shame, but not of guilt, was more reduced than in controls, only when social standards were violated, while it was not different than controls in case of moral violations. The impairment in discriminating between normal social situations and violations also emerged. Taken together, these findings suggest that the role of the amygdala in processing shame might reflect its relevance in resolving ambiguity and uncertainty, in order to correctly detect social violations and to generate shame feelings.

Roles of Category, Shape, and Spatial Frequency in Shaping Animal and Tool Selectivity in the Occipitotemporal Cortex

Does the nature of representation in the category-selective regions in the occipitotemporal cortex reflect visual or conceptual properties? Previous research showed that natural variability in visual features across categories, quantified by image gist statistics, is highly correlated with the different neural responses observed in the occipitotemporal cortex. Using fMRI, we examined whether category selectivity for animals and tools would remain, when image gist statistics were comparable across categories. Critically, we investigated how category, shape, and spatial frequency may contribute to the category selectivity in the animal- and tool-selective regions. Female and male human observers viewed low- or high-passed images of round or elongated animals and tools that shared comparable gist statistics in the main experiment, and animal and tool images of naturally varied gist statistics in a separate localizer. Univariate analysis revealed robust category-selective responses for images with comparable gist statistics across categories. Successful classification for category (animals/tools), shape (round/elongated), and spatial frequency (low/high) was also observed, with highest classification accuracy for category. Representational similarity analyses further revealed that the activation patterns in the animal-selective regions were most correlated with a model that represents only animal information, whereas the activation patterns in the tool-selective regions were most correlated with a model that represents only tool information, suggesting that these regions selectively represent information of only animals or tools. Together, in addition to visual features, the distinction between animal and tool representations in the occipitotemporal cortex is likely shaped by higher-level conceptual influences such as categorization or interpretation of visual inputs.

SIGNIFICANCE STATEMENT Since different categories often vary systematically in both visual and conceptual features, it remains unclear what kinds of information determine category-selective responses in the occipitotemporal cortex. To minimize the influences of low- and mid-level visual features, here we used a diverse image set of animals and tools that shared comparable gist statistics. We manipulated category (animals/tools), shape (round/elongated), and spatial frequency (low/high), and found that the representational content of the animal- and tool-selective regions is primarily determined by their preferred categories only, regardless of shape or spatial frequency. Our results show that category-selective responses in the occipitotemporal cortex are influenced by higher-level processing such as categorization or interpretation of visual inputs, and highlight the specificity in these category-selective regions.

A novel social attribution paradigm: The Dynamic Interacting Shape Clips (DISC)

The Dynamic Interacting Shape Clips (DISC) is a novel stimulus set designed to examine mentalizing, specifically social attribution, suitable for use with diverse methodologies including fMRI. The DISC offer some advantages compared to other social attribution stimuli including a large number of stimuli, subsets of stimuli depicting different kinds of social interactions (i.e., friendly approach, aggression, and avoidance), and two control tasks-one that contrasts interpretations of socially contingent movement versus random, inanimate movement, and the other that examines the impact of attentional shifts on mentalizing using the same visual stimuli with a different cue. This study describes both behavioral and fMRI findings from a sample of 22 typically developing adults (m_age = 21.7 years, SD = 1.72). Behavioral data supports participants anthropomorphized the stimuli and the social intent of the clips were perceived as intended. Neuroimaging findings demonstrate that brain areas associated with processing animacy and mental state attribution were activated when participants were shown clips featuring social interactions compared to random movement, and when attention was cued to social versus physical aspects of the same stimuli. Results lend empirical support for the use of the DISC in future studies of social cognition.

The Acquisition of Person Knowledge

How do we learn what we know about others? Answering this question requires understanding the perceptual mechanisms with which we recognize individuals and their actions, and the processes by which the resulting perceptual representations lead to inferences about people's mental states and traits. This review discusses recent behavioral, neural, and computational studies that have contributed to this broad research program, encompassing both social perception and social cognition.

The nature of the animacy organization in human ventral temporal cortex

The principles underlying the animacy organization of the ventral temporal cortex (VTC) remain hotly debated, with recent evidence pointing to an animacy continuum rather than a dichotomy. What drives this continuum? According to the visual categorization hypothesis, the continuum reflects the degree to which animals contain animal-diagnostic features. By contrast, the agency hypothesis posits that the continuum reflects the degree to which animals are perceived as (social) agents. Here, we tested both hypotheses with a stimulus set in which visual categorizability and agency were dissociated based on representations in convolutional neural networks and behavioral experiments. Using fMRI, we found that visual categorizability and agency explained independent components of the animacy continuum in VTC. Modeled together, they fully explained the animacy continuum. Finally, clusters explained by visual categorizability were localized posterior to clusters explained by agency. These results show that multiple organizing principles, including agency, underlie the animacy continuum in VTC.

An Information-Driven 2-Pathway Characterization of Occipitotemporal and Posterior Parietal Visual Object Representations

Recent studies have demonstrated the existence of rich visual representations in both occipitotemporal cortex (OTC) and posterior parietal cortex (PPC). Using fMRI decoding and a bottom-up data-driven approach, we showed that although robust object category representations exist in both OTC and PPC, there is an information-driven 2-pathway separation among these regions in the representational space, with occipitotemporal regions arranging hierarchically along 1 pathway and posterior parietal regions along another pathway. We obtained 10 independent replications of this 2-pathway distinction, accounting for 58-81% of the total variance of the region-wise differences in visual representation. The separation of the PPC regions from higher occipitotemporal regions was not driven by a difference in tolerance to changes in low-level visual features, did not rely on the presence of special object categories, and was present whether or not object category was task relevant. Our information-driven 2-pathway structure differs from the well-known ventral-what and dorsal-where/how characterization of posterior brain regions. Here both pathways contain rich nonspatial visual representations. The separation we see likely reflects a difference in neural coding scheme used by PPC to represent visual information compared with that of OTC.

Perceiving animacy purely from visual motion cues involves intraparietal sulcus

Distinguishing animate from inanimate objects is fundamental for social perception in humans and animals. Visual motion cues indicative of self-propelled object motion are useful for animacy perception: they can be detected over a wide expanse of visual field, at distance and in low visibility conditions, can attract attention and provide clues about object behaviour. However, the neural correlates of animacy perception evoked exclusively by visual motion cues, i.e. not relying on form, background or visual context, are unclear. We aimed to address this question in four psychophysical experiments in humans, two of which performed during neuroimaging. The stimulus was a single dot with constant form that moved on a blank background and evoked controlled degrees of perceived animacy through parametric variations of self-propelled motion cues. BOLD signals reflecting perceived animacy in a graded manner irrespective of eye movements were found in one intraparietal region. Additional whole-brain and region-of-interest analyses revealed no comparable effects in brain regions associated with social processing or other areas. Our study shows that animacy perception evoked solely by visual motion cues, a basic perceptual process in social cognition, engages brain regions not primarily associated with social cognition.

Anthropomorphizing without Social Cues Requires the Basolateral Amygdala

Anthropomorphism, the attribution of distinctively human mental characteristics to nonhuman animals and objects, illustrates the human propensity for extending social cognition beyond typical social targets. Yet, its processing components remain challenging to study because they are typically all engaged simultaneously. Across one pilot study and one focal study, we tested three rare people with basolateral amygdala lesions to dissociate two specific processing components: those triggered by attention to social cues (e.g., seeing a face) and those triggered by endogenous semantic knowledge (e.g., imbuing a machine with animacy). A pilot study demonstrated that, like neurologically intact control group participants, the three amygdala-damaged participants produced anthropomorphic descriptions for highly socially salient stimuli but not for stimuli lacking clear social cues. A focal study found that the three amygdala participants could anthropomorphize animate and living entities normally, but anthropomorphized inanimate stimuli less than control participants. Our findings suggest that the amygdala contributes to how we anthropomorphize stimuli that are not explicitly social.

Multivariate pattern analysis of the human pSTS: A comparison of three prototypical localizers

The posterior extent of the human superior temporal sulcus (pSTS) is an important cortical region for detecting animacy, attributing agency to others, and decoding goal-directed behavior. Theoretical accounts attribute these cognitive skills to unique neural populations that have been difficult to identify empirically (Hein and Knight, 2008). The aim of this study is to evaluate the multivariate statistical structure of pSTS activation patterns when viewing different social cues. We identified a core conjunction region on pSTS from univariate responses with preference for point-light biological motion, faces and the attribution of social concepts to simple animated shapes. In a multivariate analysis, we characterized the similarity structure of the resulting activation patterns after controlling for variance in the activation profile elicited by form and motion features. We found strong antagonistic activation profiles between the social conditions and their localizer controls, a harbinger of why these canonical localizers are so effective, even in individual subjects. We also found unique patterns of similarity between the three core social conditions. Our findings are consistent with the Shultz et al. (2015) model of pSTS function in which separate neural populations exist for animacy detection from body parts versus for extracting intentional cues from movement.

Neural responses to visually observed social interactions

Success in the social world requires the ability to perceive not just individuals and their actions, but pairs of people and the interactions between them. Despite the complexity of social interactions, humans are adept at interpreting those interactions they observe. Although the brain basis of this remarkable ability has remained relatively unexplored, converging functional MRI evidence suggests the posterior superior temporal sulcus (pSTS) is centrally involved. Here, we sought to determine whether this region is sensitive to both the presence of interactive information, as well as to the content of qualitatively different interactions (i.e. competition vs. cooperation). Using point-light human figure stimuli, we demonstrate that the right pSTS is maximally activated when contrasting dyadic interactions vs. dyads performing independent, non-interactive actions. We then used this task to localize the same pSTS region in an independent participant group, and tested responses to non-human moving shape stimuli (i.e. two circles’ movements conveying either interactive or non-interactive behaviour). We observed significant support vector machine classification for both the presence and type of interaction (i.e. interaction vs. non-interaction, and competition vs. cooperation, respectively) in the pSTS, as well as neighbouring temporo-parietal junction (TPJ). These findings demonstrate the important role that these regions play in perceiving and understanding social interactions, and lay the foundations for further research to fully characterize interaction responses in these areas.

•

The pSTS is sensitive to visual dynamic social interactions.

•

We show that the pSTS is sensitive to both the presence & contents of interactions.

•

This effect is independent of face & body information.

Perceiving social interactions in the posterior superior temporal sulcus

Primates are highly attuned not just to social characteristics of individual agents, but also to social interactions between multiple agents. Here we report a neural correlate of the representation of social interactions in the human brain. Specifically, we observe a strong univariate response in the posterior superior temporal sulcus (pSTS) to stimuli depicting social interactions between two agents, compared with (i) pairs of agents not interacting with each other, (ii) physical interactions between inanimate objects, and (iii) individual animate agents pursuing goals and interacting with inanimate objects. We further show that this region contains information about the nature of the social interaction-specifically, whether one agent is helping or hindering the other. This sensitivity to social interactions is strongest in a specific subregion of the pSTS but extends to a lesser extent into nearby regions previously implicated in theory of mind and dynamic face perception. This sensitivity to the presence and nature of social interactions is not easily explainable in terms of low-level visual features, attention, or the animacy, actions, or goals of individual agents. This region may underlie our ability to understand the structure of our social world and navigate within it.

The processing of animacy information is disrupted as a function of callous-unemotional traits in youth with disruptive behavior disorders

Atypical amygdala responses to emotional stimuli have been consistently reported in youth with Disruptive Behavior Disorders (DBDs; Conduct Disorder/Oppositional Defiant Disorder). However, responding to animacy stimuli has not been systematically investigated. Yet, the amygdala is known to be responsive to animacy stimuli and impairment in responsiveness to animacy information may have implications for social cognitive development. Twenty-nine youth with DBDs and 20 typically developing youth, matched for IQ, age (M_age = 14.45, SD = 2.05) and gender, completed a dot probe task during fMRI. Stimuli consisted of negative/faces, negative/objects, neutral/faces and neutral/objects images. Youth with DBDs, relative to typically developing youth, showed: i) reduced amygdala and lateral temporal cortex responses to faces relative to objects. Moreover, within the group of youth with DBDs, increasing callous-unemotional traits were associated with lesser amygdala responses to faces relative to objects. These data suggest that youth with DBDs, particularly those with high levels of CU traits exhibit dysfunction in animacy processing in the amygdala. This dysfunction may underpin the asociality reported in these youth.

•

Animacy processing within the amygdala is a critical component of social cognition.

•

Youth with DBDs had reduced responses to faces compared to objects in amygdala.

•

CU traits were associated with reduced amygdala responses to faces.

•

Youth with DBDs, particularly those with CU exhibit problems in animacy processing.

Abnormal Brain Activation During Theory of Mind Tasks in Schizophrenia: A Meta-Analysis

Social cognition abilities are severely impaired in schizophrenia (SZ). The current meta-analysis used foci of 21 individual studies on functional abnormalities in the schizophrenic brain in order to identify regions that reveal convergent under- or over-activation during theory of mind (TOM) tasks. Studies were included in the analyses when contrasting tasks that require the processing of mental states with tasks which did not. Only studies that investigated patients with an ICD or DSM diagnosis were included. Quantitative voxel-based meta-analyses were done using Seed-based d Mapping software. Common TOM regions like medial-prefrontal cortex and temporo-parietal junction revealed abnormal activation in schizophrenic patients: Under-activation was identified in the medial prefrontal cortex, left orbito-frontal cortex, and in a small section of the left posterior temporo-parietal junction. Remarkably, robust over-activation was identified in a more dorsal, bilateral section of the temporo-parietal junction. Further abnormal activation was identified in medial occipito-parietal cortex, right premotor areas, left cingulate gyrus, and lingual gyrus. The findings of this study suggest that SZ patients simultaneously show over- and under-activation in TOM-related regions. Especially interesting, temporo-parietal junction reveals diverging activation patterns with an under-activating left posterior and an over-activating bilateral dorsal section. In conclusion, SZ patients show less specialized brain activation in regions linked to TOM and increased activation in attention-related networks suggesting compensatory effects.

Neural mechanisms underlying valence inferences to sound: The role of the right angular gyrus

We frequently infer others' intentions based on non-verbal auditory cues. Although the brain underpinnings of social cognition have been extensively studied, no empirical work has yet examined the impact of musical structure manipulation on the neural processing of emotional valence during mental state inferences. We used a novel sound-based theory-of-mind paradigm in which participants categorized stimuli of different sensory dissonance level in terms of positive/negative valence. Whilst consistent with previous studies which propose facilitated encoding of consonances, our results demonstrated that distinct levels of consonance/dissonance elicited differential influences on the right angular gyrus, an area implicated in mental state attribution and attention reorienting processes. Functional and effective connectivity analyses further showed that consonances modulated a specific inhibitory interaction from associative memory to mental state attribution substrates. Following evidence suggesting that individuals with autism may process social affective cues differently, we assessed the relationship between participants' task performance and self-reported autistic traits in clinically typical adults. Higher scores on the social cognition scales of the AQ were associated with deficits in recognising positive valence in consonant sound cues. These findings are discussed with respect to Bayesian perspectives on autistic perception, which highlight a functional failure to optimize precision in relation to prior beliefs.

Neuroanatomical Substrates of Rodent Social Behavior: The Medial Prefrontal Cortex and Its Projection Patterns

Social behavior encompasses a number of distinctive and complex constructs that form the core elements of human imitative culture, mainly represented as either affiliative or antagonistic interactions with conspecifics. Traditionally considered in the realm of psychology, social behavior research has benefited from recent advancements in neuroscience that have accelerated identification of the neural systems, circuits, causative genes and molecular mechanisms that underlie distinct social cognitive traits. In this review article, I summarize recent findings regarding the neuroanatomical substrates of key social behaviors, focusing on results from experiments conducted in rodent models. In particular, I will review the role of the medial prefrontal cortex (mPFC) and downstream subcortical structures in controlling social behavior, and discuss pertinent future research perspectives.

Semantic memory deficits are associated with pica in individuals with acquired brain injury

Although pica is one of the most prominent signs in individuals with severe cognitive impairment, the mechanisms and neural basis for pica have not been well elucidated. To address this issue, patients with acquired brain injury who showed pica and hyperorality were investigated. Eleven patients with pica, i.e., individuals who eat non-food items, and eight patients with hyperorality but who never eat non-food items were recruited. The cognitive and behavioral assessments and neural substrates of the two groups were compared. For basic cognitive and behavioral functions, two kinds of mental state examination-the mini-mental state examination and the new clinical scale for rating of mental states of the elderly-were administered. For pica-related behavioral features, frontal release signs, semantic memory deficits, and changes in eating behaviors were compared. Compared with the hyperorality group, the pica group had more severe semantic memory deficits and fewer frontal release signs, whereas there was no significant difference in changes in eating behaviors. Individuals in the pica group always had a lesion in the posterior part of the middle temporal gyrus. These findings suggest that semantic memory deficits following temporal lobe damage are associated with pica.

Sensory cortical response to uncertainty and low salience during recognition of affective cues in musical intervals

Previous neuroimaging studies have shown an increased sensory cortical response (i.e., heightened weight on sensory evidence) under higher levels of predictive uncertainty. The signal enhancement theory proposes that attention improves the quality of the stimulus representation, and therefore reduces uncertainty by increasing the gain of the sensory signal. The present study employed functional magnetic resonance imaging (fMRI) to investigate the neural correlates for ambiguous valence inferences signaled by auditory information within an emotion recognition paradigm. Participants categorized sound stimuli of three distinct levels of consonance/dissonance controlled by interval content. Separate behavioural and neuroscientific experiments were conducted. Behavioural results revealed that, compared with the consonance condition (perfect fourths, fifths and octaves) and the strong dissonance condition (minor/major seconds and tritones), the intermediate dissonance condition (minor thirds) was the most ambiguous, least salient and more cognitively demanding category (slowest reaction times). The neuroscientific findings were consistent with a heightened weight on sensory evidence whilst participants were evaluating intermediate dissonances, which was reflected in an increased neural response of the right Heschl’s gyrus. The results support previous studies that have observed enhanced precision of sensory evidence whilst participants attempted to represent and respond to higher degrees of uncertainty, and converge with evidence showing preferential processing of complex spectral information in the right primary auditory cortex. These findings are discussed with respect to music-theoretical concepts and recent Bayesian models of perception, which have proposed that attention may heighten the weight of information coming from sensory channels to stimulate learning about unknown predictive relationships.

Motor experience influences object knowledge

An object's perceived readiness-for-action (e.g., its size, the degree of rotation from its canonical position, the user's viewpoint) can influence semantic knowledge retrieval. Yet, the organization of object knowledge may also be affected by body-specific sensorimotor experiences. Here, we investigated whether people's history of performing motor actions with their hands influences the knowledge they store and retrieve about graspable objects. We compared object representations between healthy right- and left-handers (Experiment 1), and between unilateral stroke patients, whose motor experience was changed by impairment of either their right or left hand (Experiment 2). Participants saw pictures of graspable everyday items with the handles oriented toward either the left or right hand, and they generated the type of grasp they would employ (i.e., clench or pinch) when using each object, responding orally. In both experiments, hand dominance and object orientation interacted to predict response times. In Experiment 1, judgments were fastest when objects were oriented toward the right hand in right-handers, but not in left-handers. In Experiment 2, judgments were fastest when objects were oriented toward the left hand in patients who had lost the use of their right hand, even though these patients were right-handed prior to brain injury. Results suggest that at least some aspects of object knowledge are determined by motor experience, and can be changed by new patterns of motor experience. People with different bodily characteristics, who interact with objects in systematically different ways, form correspondingly different neurocognitive representations of the same common objects. (PsycINFO Database Record

Representational Similarity Mapping of Distributional Semantics in Left Inferior Frontal, Middle Temporal, and Motor Cortex

Language comprehension engages a distributed network of frontotemporal, parietal, and sensorimotor regions, but it is still unclear how meaning of words and their semantic relationships are represented and processed within these regions and to which degrees lexico-semantic representations differ between regions and semantic types. We used fMRI and representational similarity analysis to relate word-elicited multivoxel patterns to semantic similarity between action and object words. In left inferior frontal (BA 44-45-47), left posterior middle temporal and left precentral cortex, the similarity of brain response patterns reflected semantic similarity among action-related verbs, as well as across lexical classes-between action verbs and tool-related nouns and, to a degree, between action verbs and food nouns, but not between action verbs and animal nouns. Instead, posterior inferior temporal cortex exhibited a reverse response pattern, which reflected the semantic similarity among object-related nouns, but not action-related words. These results show that semantic similarity is encoded by a range of cortical areas, including multimodal association (e.g., anterior inferior frontal, posterior middle temporal) and modality-preferential (premotor) cortex and that the representational geometries in these regions are partly dependent on semantic type, with semantic similarity among action-related words crossing lexical-semantic category boundaries.

Different aberrant mentalizing networks in males and females with autism spectrum disorders: Evidence from resting-state functional magnetic resonance imaging

Previous studies have found that individuals with autism spectrum disorders show impairments in mentalizing processes and aberrant brain activity compared with typically developing participants. However, the findings are mainly from male participants and the aberrant effects in autism spectrum disorder females and sex differences are still unclear. To address these issues, this study analyzed intrinsic functional connectivity of mentalizing regions using resting-state functional magnetic resonance imaging data of 48 autism spectrum disorder males and females and 48 typically developing participants in autism brain imaging data exchange. Whole-brain analyses showed that autism spectrum disorder males had hyperconnectivity in functional connectivity of the bilateral temporal-parietal junction, whereas autism spectrum disorder females showed hypoconnectivity in functional connectivity of the medial prefrontal cortex, precuneus, and right temporal-parietal junction. Interaction between sex and autism was found in both short- and long-distance functional connectivity effects, confirming that autism spectrum disorder males showed overconnectivity, while autism spectrum disorder females showed underconnectivity. Furthermore, a regression analysis revealed that in autism spectrum disorder, males and females demonstrated different relations between the functional connectivity effects of the mentalizing regions and the core autism spectrum disorder deficits. These results suggest sex differences in the mentalizing network in autism spectrum disorder individuals. Future work is needed to examine how sex interacts with other factors such as age and the sex differences during mentalizing task performance.