The role of motion in the neural representation of social interactions in the posterior temporal cortex

Mind in others' shoes: Neuroscientific protocol for external referent decision awareness (ERDA) in organizations

This study investigates the neural and physiological mechanisms underlying External Referent Decision Awareness (ERDA) within organizational contexts, focusing on hierarchical roles (Head, Peer, Staff). Twenty-two professionals participated, and electroencephalographic (EEG frequency band: Delta, Theta, Alpha, Beta, Gamma) and autonomic indices (skin conductance and cardiovascular indices) were recorded, while personality traits and decision-making styles were assessed. Results revealed higher Delta and Theta activation in the left temporo-parietal junction (TPJ) during Peer-related decisions, reflecting increased social cognition and ambiguity regulation in those contexts. Gamma activity, associated with high-order cognitive processes, was prominent in the left frontal cortex across all roles, indicating complex decision evaluation. These findings underscore the complexity of low-frequency bands (Delta and Theta), involved in emotional regulation and social cognition, while high-frequency bands (Gamma) reflect cognitive integration and decision complexity. Furthermore, autonomic data showed higher Skin Conductance Levels (SCL) for Head decisions, suggesting greater emotional involvement.The findings revealed a significant negative correlation between avoidant decision-making styles and the neural and behavioral evaluations of leader decisions, suggesting reduced engagement of neurocognitive systems involved in reward processing and evaluative judgment in individuals with a tendency to avoid decision-making. Additionally, higher extraversion correlated with more favorable evaluations of decisions made by Staff, potentially indicating greater activation in neural circuits associated with social reward and group dynamics. In conclusion, these findings suggest that neural activity and personality traits interact to shape hierarchical decision-making awareness, highlighting the need for tailored leadership and decision-making strategies in organizations.

COMMON AND DISTINCT NEURAL CORRELATES OF SOCIAL INTERACTION PERCEPTION AND THEORY OF MIND

Social cognition spans from perceiving agents and their interactions to making inferences based on theory of mind (ToM). Despite their frequent co-occurrence in real life, the commonality and distinction between social interaction perception and ToM at behavioral and neural levels remain unclear. Here, participants (N = 231) provided moment-by-moment ratings of four text and four audio narratives on social interactions and ToM engagement. Social interaction and ToM ratings were reliable (split-half r = .98 and .92, respectively) but only modestly correlated across time (r = .32). In a second sample (N = 90), we analyzed co-variation between normative social interaction and ToM ratings and functional magnetic resonance (fMRI) activity during narrative reading (text) and listening (audio). Social interaction perception and ToM activity maps generalized across text and audio presentation (r = .83 and .57 between unthresholded t maps, respectively). When ToM was held constant, merely perceiving social interactions activated all regions canonically associated with ToM under both modalities (FDR q < .01), including temporoparietal junction, superior temporal sulcus, medial prefrontal cortex, and precuneus. ToM activated these regions as well, indicating a shared, modality-general system for social interaction perception and ToM. Furthermore, ToM uniquely engaged lateral occipitotemporal cortex, left anterior intraparietal sulcus, and right premotor cortex. These results imply that perceiving social interactions automatically engages regions implicated in mental state inferences. In addition, ToM is distinct from social interaction perception in its recruitment of regions associated with higher-level cognitive processes, including action understanding and executive functions.

Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision

We effortlessly extract behaviorally relevant information from dynamic visual input in order to understand the actions of others. In the current study, we develop and test a number of models to better understand the neural representational geometries supporting action understanding. Using fMRI, we measured brain activity as participants viewed a diverse set of 90 different video clips depicting social and nonsocial actions in real-world contexts. We developed five behavioral models using arrangement tasks: two models reflecting behavioral judgments of the purpose (transitivity) and the social content (sociality) of the actions depicted in the video stimuli; and three models reflecting behavioral judgments of the visual content (people, objects, and scene) depicted in still frames of the stimuli. We evaluated how well these models predict neural representational geometry and tested them against semantic models based on verb and nonverb embeddings and visual models based on gaze and motion energy. Our results revealed that behavioral judgments of similarity better reflect neural representational geometry than semantic or visual models throughout much of cortex. The sociality and transitivity models in particular captured a large portion of unique variance throughout the action observation network, extending into regions not typically associated with action perception, like ventral temporal cortex. Overall, our findings expand the action observation network and indicate that the social content and purpose of observed actions are predominant in cortical representation.

A quantitative comparison of atlas parcellations on the human superior temporal sulcus

The superior temporal sulcus (STS) has a functional topography that has been difficult to characterize through traditional approaches. Automated atlas parcellations may be one solution while also being beneficial for both dimensional reduction and standardizing regions of interest, but they yield very different boundary definitions along the STS. Here we evaluate how well machine learning classifiers can correctly identify six social cognitive tasks from STS activation patterns dimensionally reduced using four popular atlases (Glasser et al., 2016; Gordon et al., 2016; Power et al., 2011 as projected onto the surface by Arslan et al., 2018; Schaefer et al., 2018). Functional data was summarized within each STS parcel in one of four ways, then subjected to leave-one-subject-out cross-validation SVM classification. We found that the classifiers could readily label conditions when data was parcellated using any of the four atlases, evidence that dimensional reduction to parcels did not compromise functional fingerprints. Mean activation for the social conditions was the most effective metric for classification in the right STS, whereas all the metrics classified equally well in the left STS. Interestingly, even atlases constructed from random parcellation schemes (null atlases) classified the conditions with high accuracy. We therefore conclude that the complex activation maps on the STS are readily differentiated at a coarse granular level, despite a strict topography having not yet been identified. Further work is required to identify what features have greatest potential to improve the utility of atlases in replacing functional localizers.

Neural Mechanisms of Social Interaction Perception: Observing Interpersonal Synchrony Modulates Action Observation Network Activation and Is Spared in Autism

How the temporal dynamics of social interactions are perceived arguably plays an important role in how one engages in social interactions and how difficulties in establishing smooth social interactions may occur. One aspect of temporal dynamics in social interactions is the mutual coordination of individuals' behaviors during social interaction, otherwise known as behavioral interpersonal synchrony (IPS). Behavioral IPS has been studied increasingly in various contexts, such as a feature of the social interaction difficulties inherent to autism. To fully understand the temporal dynamics of social interactions, or reductions thereof in autism, the neural basis of IPS perception needs to be established. Thus, the current study's aim was twofold: to establish the basic neuro-perceptual processing of IPS in social interactions for typical observers and to test whether it might differ for autistic individuals. In a task-based fMRI paradigm, participants viewed short, silent video vignettes of humans during social interactions featuring a variation of behavioral IPS. The results show that observing behavioral IPS modulates the Action Observation Network (AON). Interestingly, autistic participants showed similar neural activation patterns as non-autistic participants which were modulated by the behavioral IPS they observed in the videos, suggesting that the perception of temporal dynamics of social interactions is spared and may not underly reduced behavioral IPS often observed in autism. Nevertheless, a general difference in processing social interactions was found in autistic observers, characterized by decreased neural activation in the right middle frontal gyrus, angular gyrus, and superior temporal areas. These findings demonstrate that although the autistic and non-autistic groups indeed differed in the neural processing of social interaction perception, the temporal dynamics of these social interactions were not the reason for these differences in social interaction perception in autism. Hence, spared recruitment of the AON for processing temporal dynamics of social interactions in autism does not account for the widely reported attenuation of IPS in autism and for the widely reported and presently observed differences in social interaction perception in autism.

Beyond human-likeness: Socialness is more influential when attributing mental states to robots

We sought to replicate and expand previous work showing that the more human-like a robot appears, the more willing people are to attribute mind-like capabilities and socially engage with it. Forty-two participants played games against a human, a humanoid robot, a mechanoid robot, and a computer algorithm while undergoing functional neuroimaging. We confirmed that the more human-like the agent, the more participants attributed a mind to them. However, exploratory analyses revealed that the perceived socialness of an agent appeared to be as, if not more, important for mind attribution. Our findings suggest top-down knowledge cues may be equally or possibly more influential than bottom-up stimulus cues when exploring mind attribution in non-human agents. While further work is now required to test this hypothesis directly, these preliminary findings hold important implications for robotic design and to understand and test the flexibility of human social cognition when people engage with artificial agents.

Neural Representations of Observed Interpersonal Synchrony/Asynchrony in the Social Perception Network

The visual perception of individuals is thought to be mediated by a network of regions in the occipitotemporal cortex that supports specialized processing of faces, bodies, and actions. In comparison, we know relatively little about the neural mechanisms that support the perception of multiple individuals and the interactions between them. The present study sought to elucidate the visual processing of social interactions by identifying which regions of the social perception network represent interpersonal synchrony. In an fMRI study with 32 human participants (26 female, 6 male), we used multivoxel pattern analysis to investigate whether activity in face-selective, body-selective, and interaction-sensitive regions across the social perception network supports the decoding of synchronous versus asynchronous head-nodding and head-shaking. Several regions were found to support significant decoding of synchrony/asynchrony, including extrastriate body area (EBA), face-selective and interaction-sensitive mid/posterior right superior temporal sulcus, and occipital face area. We also saw robust cross-classification across actions in the EBA, suggestive of movement-invariant representations of synchrony/asynchrony. Exploratory whole-brain analyses also identified a region of the right fusiform cortex that responded more strongly to synchronous than to asynchronous motion. Critically, perceiving interpersonal synchrony/asynchrony requires the simultaneous extraction and integration of dynamic information from more than one person. Hence, the representation of synchrony/asynchrony cannot be attributed to augmented or additive processing of individual actors. Our findings therefore provide important new evidence that social interactions recruit dedicated visual processing within the social perception network that extends beyond that engaged by the faces and bodies of the constituent individuals.

Category-Selective Representation of Relationships in the Visual Cortex

Understanding social interaction requires processing social agents and their relationships. The latest results show that much of this process is visually solved: visual areas can represent multiple people encoding emergent information about their interaction that is not explained by the response to the individuals alone. A neural signature of this process is an increased response in visual areas, to face-to-face (seemingly interacting) people, relative to people presented as unrelated (back-to-back). This effect highlighted a network of visual areas for representing relational information. How is this network organized? Using functional MRI, we measured the brain activity of healthy female and male humans (N = 42), in response to images of two faces or two (head-blurred) bodies, facing toward or away from each other. Taking the facing > non-facing effect as a signature of relation perception, we found that relations between faces and between bodies were coded in distinct areas, mirroring the categorical representation of faces and bodies in the visual cortex. Additional analyses suggest the existence of a third network encoding relations between (nonsocial) objects. Finally, a separate occipitotemporal network showed the generalization of relational information across body, face, and nonsocial object dyads (multivariate pattern classification analysis), revealing shared properties of relations across categories. In sum, beyond single entities, the visual cortex encodes the relations that bind multiple entities into relationships; it does so in a category-selective fashion, thus respecting a general organizing principle of representation in high-level vision. Visual areas encoding visual relational information can reveal the processing of emergent properties of social (and nonsocial) interaction, which trigger inferential processes.

Disrupted third visual pathway function in schizophrenia: Evidence from real and implied motion processing

Impaired motion perception in schizophrenia has been associated with deficits in social-cognitive processes and with reduced activation of visual sensory regions, including the middle temporal area (MT+) and posterior superior temporal sulcus (pSTS). These findings are consistent with the recent proposal of the existence of a specific 'third visual pathway' specialized for social perception in which motion is a fundamental component. The third visual pathway transmits visual information from early sensory visual processing areas to the STS, with MT+ acting as a critical intermediary. We used functional magnetic resonance imaging to investigate functioning of this pathway during processing of naturalistic videos with explicit (real) motion and static images with implied motion cues. These measures were related to face emotion recognition and motion-perception, as measured behaviorally. Participants were 28 individuals with schizophrenia (Sz) and 20 neurotypical controls. Compared to controls, individuals with Sz showed reduced activation of third visual pathway regions (MT+, pSTS) in response to both real- and implied-motion stimuli. Dysfunction of early visual cortex and pulvinar were also associated with aberrant real-motion processing. Implied-motion stimuli additionally engaged a wide network of brain areas including parietal, motor and frontal nodes of the human mirror neuron system. The findings support concepts of MT+ as a mediator between visual sensory areas and higher-order brain and argue for greater focus on MT+ contributions to social-cognitive processing, in addition to its well-documented role in visual motion processing.

Visual neuroscience: A brain area tuned for processing social interactions

Socialising with others is part of everyday life. A new study demonstrates that a brain area specialised for visual body perception is attuned to processing social interactions between two people. Intriguingly, this area is lateralised in the left hemisphere.

Converging evidence that left extrastriate body area supports visual sensitivity to social interactions

Navigating our complex social world requires processing the interactions we observe. Recent psychophysical and neuroimaging studies provide parallel evidence that the human visual system may be attuned to efficiently perceive dyadic interactions. This work implies, but has not yet demonstrated, that activity in body-selective cortical regions causally supports efficient visual perception of interactions. We adopt a multi-method approach to close this important gap. First, using a large fMRI dataset (n = 92), we found that the left hemisphere extrastriate body area (EBA) responds more to face-to-face than non-facing dyads. Second, we replicated a behavioral marker of visual sensitivity to interactions: categorization of facing dyads is more impaired by inversion than non-facing dyads. Third, in a pre-registered experiment, we used fMRI-guided transcranial magnetic stimulation to show that online stimulation of the left EBA, but not a nearby control region, abolishes this selective inversion effect. Activity in left EBA, thus, causally supports the efficient perception of social interactions.

Hierarchical organization of social action features along the lateral visual pathway

Recent theoretical work has argued that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. Like visual representations in the ventral and dorsal streams, representations in the lateral stream are thought to be hierarchically organized. However, no prior studies have comprehensively investigated the organization of naturalistic, social visual content in the lateral stream. To address this question, we curated a naturalistic stimulus set of 250 3-s videos of two people engaged in everyday actions. Each clip was richly annotated for its low-level visual features, mid-level scene and object properties, visual social primitives (including the distance between people and the extent to which they were facing), and high-level information about social interactions and affective content. Using a condition-rich fMRI experiment and a within-subject encoding model approach, we found that low-level visual features are represented in early visual cortex (EVC) and middle temporal (MT) area, mid-level visual social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Communicative interactions, in particular, explained unique variance in regions of the STS after accounting for variance explained by all other labeled features. Taken together, these results provide support for representation of increasingly abstract social visual content-consistent with hierarchical organization-along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of the lateral visual pathway.

Seeing social interactions

Seeing the interactions between other people is a critical part of our everyday visual experience, but recognizing the social interactions of others is often considered outside the scope of vision and grouped with higher-level social cognition like theory of mind. Recent work, however, has revealed that recognition of social interactions is efficient and automatic, is well modeled by bottom-up computational algorithms, and occurs in visually-selective regions of the brain. We review recent evidence from these three methodologies (behavioral, computational, and neural) that converge to suggest the core of social interaction perception is visual. We propose a computational framework for how this process is carried out in the brain and offer directions for future interdisciplinary investigations of social perception.

Auditory dyadic interactions through the "eye" of the social brain: How visual is the posterior STS interaction region?

Human interactions contain potent social cues that meet not only the eye but also the ear. Although research has identified a region in the posterior superior temporal sulcus as being particularly sensitive to visually presented social interactions (SI-pSTS), its response to auditory interactions has not been tested. Here, we used fMRI to explore brain response to auditory interactions, with a focus on temporal regions known to be important in auditory processing and social interaction perception. In Experiment 1, monolingual participants listened to two-speaker conversations (intact or sentence-scrambled) and one-speaker narrations in both a known and an unknown language. Speaker number and conversational coherence were explored in separately localised regions-of-interest (ROI). In Experiment 2, bilingual participants were scanned to explore the role of language comprehension. Combining univariate and multivariate analyses, we found initial evidence for a heteromodal response to social interactions in SI-pSTS. Specifically, right SI-pSTS preferred auditory interactions over control stimuli and represented information about both speaker number and interactive coherence. Bilateral temporal voice areas (TVA) showed a similar, but less specific, profile. Exploratory analyses identified another auditory-interaction sensitive area in anterior STS. Indeed, direct comparison suggests modality specific tuning, with SI-pSTS preferring visual information while aSTS prefers auditory information. Altogether, these results suggest that right SI-pSTS is a heteromodal region that represents information about social interactions in both visual and auditory domains. Future work is needed to clarify the roles of TVA and aSTS in auditory interaction perception and further probe right SI-pSTS interaction-selectivity using non-semantic prosodic cues.

Intermodulation responses show integration of interacting bodies in a new whole

People are often seen among other people, relating to and interacting with one another. Recent studies suggest that socially relevant spatial relations between bodies, such as the face-to-face positioning, or facingness, change the visual representation of those bodies, relative to when the same items appear unrelated (e.g., back-to-back) or in isolation. The current study addresses the hypothesis that face-to-face bodies give rise to a new whole, an integrated representation of individual bodies in a new perceptual unit. Using frequency-tagging EEG, we targeted, as a measure of integration, an EEG correlate of the non-linear combination of the neural responses to each of two individual bodies presented either face-to-face as if interacting, or back-to-back. During EEG recording, participants (N = 32) viewed two bodies, either face-to-face or back-to-back, flickering at two different frequencies (F1 and F2), yielding two distinctive responses in the EEG signal. Spectral analysis examined the responses at the intermodulation frequencies (nF1±mF2), signaling integration of individual responses. An anterior intermodulation response was observed for face-to-face bodies, but not for back-to-back bodies, nor for face-to-face chairs and machines. These results show that interacting bodies are integrated into a representation that is more than the sum of its parts. This effect, specific to body dyads, may mark an early step in the transformation towards an integrated representation of a social event, from the visual representation of individual participants in that event.

Specialized Networks for Social Cognition in the Primate Brain

Primates have evolved diverse cognitive capabilities to navigate their complex social world. To understand how the brain implements critical social cognitive abilities, we describe functional specialization in the domains of face processing, social interaction understanding, and mental state attribution. Systems for face processing are specialized from the level of single cells to populations of neurons within brain regions to hierarchically organized networks that extract and represent abstract social information. Such functional specialization is not confined to the sensorimotor periphery but appears to be a pervasive theme of primate brain organization all the way to the apex regions of cortical hierarchies. Circuits processing social information are juxtaposed with parallel systems involved in processing nonsocial information, suggesting common computations applied to different domains. The emerging picture of the neural basis of social cognition is a set of distinct but interacting subnetworks involved in component processes such as face perception and social reasoning, traversing large parts of the primate brain.

Alternative Brain Connectivity Underscores Age-Related Differences in the Processing of Interactive Biological Motion

Rapidly recognizing and understanding others' social interactions is an important ability that relies on deciphering multiple sources of information, for example, perceiving body information and inferring others' intentions. Despite recent advances in characterizing the brain basis of this ability in adults, its developmental underpinnings are virtually unknown. Here, we used fMRI to investigate which sources of social information support superior temporal sulcus responses to interactive biological motion (i.e., 2 interacting point-light human figures) at different developmental intervals in human participants (of either sex): Children show supportive functional connectivity with key nodes of the mentalizing network, while adults show stronger reliance on regions associated with body- and dynamic social interaction/biological motion processing. We suggest that adults use efficient action-intention understanding via body and biological motion information, while children show a stronger reliance on hidden mental state inferences as a potential means of learning to better understand others' interactive behavior.SIGNIFICANCE STATEMENT Recognizing others' interactive behavior is a critical human skill that depends on different sources of social information (e.g., observable body-action information, inferring others' hidden mental states, etc.). Understanding the brain-basis of this ability and characterizing how it emerges across development are important goals in social neuroscience. Here, we used fMRI to investigate which sources of social information support interactive biological motion processing in children (6-12 years) and adults. These results reveal a striking developmental difference in terms of how wider-brain connectivity shapes functional responses to interactive biological motion that suggests a reliance on distinct neuro-cognitive strategies in service of interaction understanding (i.e., children and adults show a greater reliance on explicit and implicit intentional inference, respectively).

Multimodal processing in face-to-face interactions: A bridging link between psycholinguistics and sensory neuroscience

In face-to-face communication, humans are faced with multiple layers of discontinuous multimodal signals, such as head, face, hand gestures, speech and non-speech sounds, which need to be interpreted as coherent and unified communicative actions. This implies a fundamental computational challenge: optimally binding only signals belonging to the same communicative action while segregating signals that are not connected by the communicative content. How do we achieve such an extraordinary feat, reliably, and efficiently? To address this question, we need to further move the study of human communication beyond speech-centred perspectives and promote a multimodal approach combined with interdisciplinary cooperation. Accordingly, we seek to reconcile two explanatory frameworks recently proposed in psycholinguistics and sensory neuroscience into a neurocognitive model of multimodal face-to-face communication. First, we introduce a psycholinguistic framework that characterises face-to-face communication at three parallel processing levels: multiplex signals, multimodal gestalts and multilevel predictions. Second, we consider the recent proposal of a lateral neural visual pathway specifically dedicated to the dynamic aspects of social perception and reconceive it from a multimodal perspective ("lateral processing pathway"). Third, we reconcile the two frameworks into a neurocognitive model that proposes how multiplex signals, multimodal gestalts, and multilevel predictions may be implemented along the lateral processing pathway. Finally, we advocate a multimodal and multidisciplinary research approach, combining state-of-the-art imaging techniques, computational modelling and artificial intelligence for future empirical testing of our model.

Seeing Social: A Neural Signature for Conscious Perception of Social Interactions

Social information is some of the most ambiguous content we encounter in our daily lives, yet in experimental contexts, percepts of social interactions-that is, whether an interaction is present and if so, the nature of that interaction-are often dichotomized as correct or incorrect based on experimenter-assigned labels. Here, we investigated the behavioral and neural correlates of subjective (or conscious) social perception using data from the Human Connectome Project in which participants (n = 1049; 486 men, 562 women) viewed animations of geometric shapes during fMRI and indicated whether they perceived a social interaction or random motion. Critically, rather than experimenter-assigned labels, we used observers' own reports of "Social" or "Non-social" to classify percepts and characterize brain activity, including leveraging a particularly ambiguous animation perceived as "Social" by some but "Non-social" by others to control for visual input. Behaviorally, observers were biased toward perceiving information as social (vs non-social); and neurally, observer reports (compared with experimenter labels) explained more variance in activity across much of the brain. Using "Unsure" reports, we identified several regions that responded parametrically to perceived socialness. Neural responses to social versus non-social content diverged early in time and in the cortical hierarchy. Finally, individuals with higher internalizing trait scores showed both a higher response bias toward "Social" and an inverse relationship with activity in default mode and visual association areas while scanning for social information. Findings underscore the subjective nature of social perception and the importance of using observer reports to study percepts of social interactions.SIGNIFICANCE STATEMENT Simple animations involving two or more geometric shapes have been used as a gold standard to understand social cognition and impairments therein. Yet, experimenter-assigned labels of what is social versus non-social are frequently used as a ground truth, despite the fact that percepts of such ambiguous social stimuli are highly subjective. Here, we used behavioral and fMRI data from a large sample of neurotypical individuals to show that participants' responses reveal subtle behavioral biases, help us study neural responses to social content more precisely, and covary with internalizing trait scores. Our findings underscore the subjective nature of social perception and the importance of considering observer reports in studying behavioral and neural dynamics of social perception.