Social cognition in context: A naturalistic imaging approach

"All the Stars Will Be Wells with a Rusty Pulley": Neural Processing of the Social and Pragmatic Content in a Narrative

Making sense of natural language and narratives requires building and manipulating a situation model by adding incoming information to the model and using the context stored in the model to comprehend subsequent details and events. Situation model maintenance is supported by the default mode network (DMN), but comprehension of the individual moments in the narrative relies on access to the conceptual store within the semantic system. The present study examined how these systems are engaged by different narrative content to investigate whether highly informative, or semantic, content is a particularly strong driver of semantic system activation compared with contextually driven content that requires using the situation model, which might instead engage DMN regions. The study further investigated which subregions of the graded semantic hub in the left anterior temporal lobe (ATL) were engaged by the type of narrative content. To do this, we quantified the semantic, pragmatic, social, ambiguous, and emotional content for each sentence in a complete narrative, the English translation of The Little Prince. Increased activation in the transmodal hub in the ventral ATL was only observed for high semantic (i.e., informative) relative to low semantic sentences. Activation in the dorsolateral and ventrolateral ATL subregions was observed for both high relative to low semantic and social content sentences, but the ventrolateral ATL effects were more extensive in the social condition. There was high correspondence between the social and pragmatic content results, particularly in the ventrolateral ATL. We argue that the ventrolateral ATL may be particularly engaged by internal, or endogenous, processing demands, aided by functional connections between the anterior middle temporal gyrus and the DMN. Pragmatic and social content may have driven endogenous processing given the pervasive and plot-progressing nature of this content in the narrative. We put forward a revised account of how the semantic system is engaged in naturalistic contexts, a critical step toward better understanding real-world semantic and social processing.

Shared representations of human actions across vision and language

Humans can recognize and communicate about many actions performed by others. How are actions organized in the mind, and is this organization shared across vision and language? We collected similarity judgments of human actions depicted through naturalistic videos and sentences, and tested four models of action categorization, defining actions at different levels of abstraction ranging from specific (action verb) to broad (action target: whether an action is directed towards an object, another person, or the self). The similarity judgments reflected a shared organization of action representations across videos and sentences, determined mainly by the target of actions, even after accounting for other semantic features. Furthermore, language model embeddings predicted the behavioral similarity of action videos and sentences, and captured information about the target of actions alongside unique semantic information. Together, our results show that action concepts are similarly organized in the mind across vision and language, and that this organization reflects socially relevant goals.

Reexamining the Kuleshov effect: Behavioral and neural evidence from authentic film experiments

Film cognition explores the influence of cinematic elements, such as editing and film color, on viewers' perception. The Kuleshov effect, a famous example of how editing influences viewers' emotional perception, was initially proposed to support montage theory through the Kuleshov experiment. This effect, which has since been recognized as a manifestation of point-of-view (POV) editing practices, posits that the emotional interpretation of neutral facial expressions is influenced by the accompanying emotional scene in a face-scene-face sequence. However, concerns persist regarding the validity of previous studies, often employing inauthentic film materials like static images, leaving the question of its existence in authentic films unanswered. This study addresses these concerns by utilizing authentic films in two experiments. In Experiment 1, multiple film clips were captured under the guidance of a professional film director and seamlessly integrated into authentic film sequences. 59 participants viewed these face-scene-face film sequences and were tasked with rating the valence and emotional intensity of neutral faces. The findings revealed that the accompanying fearful or happy scenes significantly influence the interpretation of emotion on neutral faces, eliciting perceptions of negative or positive emotions from the neutral face. These results affirm the existence of the Kuleshov effect within authentic films. In Experiment 2, 31 participants rated the valence and arousal of neutral faces while undergoing functional magnetic resonance imaging (fMRI). The behavioral results confirm the Kuleshov effect in the MRI scanner, while the neural data identify neural correlates that support its existence at the neural level. These correlates include the cuneus, precuneus, hippocampus, parahippocampal gyrus, post cingulate gyrus, orbitofrontal cortex, fusiform gyrus, and insula. These findings also underscore the contextual framing inherent in the Kuleshov effect. Overall, the study integrates film theory and cognitive neuroscience experiments, providing robust evidence supporting the existence of the Kuleshov effect through both subjective ratings and objective neuroimaging measurements. This research also contributes to a deeper understanding of the impact of film editing on viewers' emotional perception from the contemporary POV editing practices and neurocinematic perspective, advancing the knowledge of film cognition.

Human reasoning on social interactions in ecological contexts: insights from the theory of mind brain circuits

Introduction

The relationship between neural social cognition patterns and performance on social cognition tasks in daily life is a topic of debate, with key consideration given to the extent to which theory of mind (ToM) brain circuits share properties reflecting everyday social functioning. To test the efficacy of ecological stimuli in eliciting brain activation within the ToM brain circuits, we adapted the Edinburgh Social Cognition test social scenarios, consisting of dynamic ecological contextually embedded social stimuli, to a fMRI paradigm.

Methods

Forty-two adults (21 men, mean age ± SD = 34.19 years ±12.57) were enrolled and underwent an fMRI assessment which consisted of a ToM task using the Edinburgh Social Cognition test scenarios. We used the same stimuli to prompt implicit (movie viewing) and explicit (silent and two-choice answers) reasoning on cognitive and affective mental states. The fMRI analysis was based on the classical random effect analysis. Group inferences were complemented with supplemental analyses using overlap maps to assess inter-subject variability.

Results

We found that explicit mentalizing reasoning yielded wide neural activations when two-choice answers were used. We also observed that the nature of ToM reasoning, that is, affective or cognitive, played a significant role in activating different neural circuits.

Discussion

The ESCoT stimuli were particularly effective in evoking ToM core neural underpinnings and elicited executive frontal loops. Future work may employ the task in a clinical setting to investigate ToM network reorganization and plasticity.

Familiarity enhances functional connectivity between visual and nonvisual regions of the brain during natural viewing

We explored the neural correlates of familiarity with people and places using a naturalistic viewing paradigm. Neural responses were measured using functional magnetic resonance imaging, while participants viewed a movie taken from Game of Thrones. We compared inter-subject correlations and functional connectivity in participants who were either familiar or unfamiliar with the TV series. Higher inter-subject correlations were found between familiar participants in regions, beyond the visual brain, that are typically associated with the processing of semantic, episodic, and affective information. However, familiarity also increased functional connectivity between face and scene regions in the visual brain and the nonvisual regions of the familiarity network. To determine whether these regions play an important role in face recognition, we measured responses in participants with developmental prosopagnosia (DP). Consistent with a deficit in face recognition, the effect of familiarity was significantly attenuated across the familiarity network in DP. The effect of familiarity on functional connectivity between face regions and the familiarity network was also attenuated in DP. These results show that the neural response to familiarity involves an extended network of brain regions and that functional connectivity between visual and nonvisual regions of the brain plays an important role in the recognition of people and places during natural viewing.

The relationship between the resting state functional connectivity and social cognition in schizophrenia: Results from the Italian Network for Research on Psychoses

Deficits in social cognition (SC) interfere with recovery in schizophrenia (SZ) and may be related to resting state brain connectivity. This study aimed at assessing the alterations in the relationship between resting state functional connectivity and the social-cognitive abilities of patients with SZ compared to healthy subjects. We divided the brain into 246 regions of interest (ROI) following the Human Healthy Volunteers Brainnetome Atlas. For each participant, we calculated the resting-state functional connectivity (rsFC) in terms of degree centrality (DC), which evaluates the total strength of the most powerful coactivations of every ROI with all other ROIs during rest. The rs-DC of the ROIs was correlated with five measures of SC assessing emotion processing and mentalizing in 45 healthy volunteers (HVs) chosen as a normative sample. Then, controlling for symptoms severity, we verified whether these significant associations were altered, i.e., absent or of opposite sign, in 55 patients with SZ. We found five significant differences between SZ patients and HVs: in the patients' group, the correlations between emotion recognition tasks and rsFC of the right entorhinal cortex (R-EC), left superior parietal lobule (L-SPL), right caudal hippocampus (R-c-Hipp), and the right caudal (R-c) and left rostral (L-r) middle temporal gyri (MTG) were lost. An altered resting state functional connectivity of the L-SPL, R-EC, R-c-Hipp, and bilateral MTG in patients with SZ may be associated with impaired emotion recognition. If confirmed, these results may enhance the development of non-invasive brain stimulation interventions targeting those cerebral regions to reduce SC deficit in SZ.

Dynamic and/or multimodal assessments for social cognition in neuropsychology: Results from a systematic literature review

Objective: Despite the prevalence of socio-cognitive disturbances, and their important diagnostic/therapeutic implications, the assessment of these disturbances remains scarce. This systematic review aims to identify available social cognition tools for adult assessment that use multimodal and/or dynamic social cues, specifying their strengths and limitations (e.g. from a methodological, psychometric, ecological, and clinical perspective). Method: An electronic search was conducted in Pubmed, PsychINFO, Embase and Scopus databases for articles published up to the 3th of January 2023 and the first 200 Google Scholar results on the same date. The PRISMA methodology was applied, 3884 studies were screened based on title and abstract and 329 full texts were screened. Articles using pseudo-dynamic methodologies (e.g. morphing), reported only subjective or self-reported measures, or investigated only physiological or brain activity responses were excluded. Results: In total, 149 works were included in this review, representing 65 assessment tools (i.e. 48% studying emotion recognition (n = 31), 32% Theory of Mind (n = 21), 5% empathy (n = 3), 1.5% moral cognition/social reasoning (n = 1), and 14% being multimodal (n = 9)). For each study, the tool's main characteristics, psychometric properties, ecological validity indicators and available norms are reported. The tools are presented according to social-cognitive process assessed and communication channels used. Conclusions: This study highlights the lack of validated and standardized tools. A few tools appear to partially meet some clinical needs. The development of methodologies using a first-person paradigm and taking into account the multidimensional nature of social cognition seems a relevant research endeavour for greater ecological validity.

A shared neural code for perceiving and remembering social interactions in the human superior temporal sulcus

Recognizing and remembering social information is a crucial cognitive skill. Neural patterns in the superior temporal sulcus (STS) support our ability to perceive others' social interactions. However, despite the prominence of social interactions in memory, the neural basis of remembering social interactions is still unknown. To fill this gap, we investigated the brain mechanisms underlying memory of others' social interactions during free spoken recall of a naturalistic movie. By applying machine learning-based fMRI encoding analyses to densely labeled movie and recall data we found that a subset of the STS activity evoked by viewing social interactions predicted neural responses in not only held-out movie data, but also during memory recall. These results provide the first evidence that activity in the STS is reinstated in response to specific social content and that its reactivation underlies our ability to remember others' interactions. These findings further suggest that the STS contains representations of social interactions that are not only perceptually driven, but also more abstract or conceptual in nature.

Development and validation of the Emotional Climate Change Stories (ECCS) stimuli set

Climate change is widely recognised as an urgent issue, and the number of people concerned about it is increasing. While emotions are among the strongest predictors of behaviour change in the face of climate change, researchers have only recently begun to investigate this topic experimentally. This may be due to the lack of standardised, validated stimuli that would make studying such a topic in experimental settings possible. Here, we introduce a novel Emotional Climate Change Stories (ECCS) stimuli set. ECCS consists of 180 realistic short stories about climate change, designed to evoke five distinct emotions-anger, anxiety, compassion, guilt and hope-in addition to neutral stories. The stories were created based on qualitative data collected in two independent studies: one conducted among individuals highly concerned about climate change, and another one conducted in the general population. The stories were rated on the scales of valence, arousal, anger, anxiety, compassion, guilt and hope in the course of three independent studies. First, we explored the underlying structure of ratings (Study 1; n = 601). Then we investigated the replicability (Study 2; n = 307) and cross-cultural validity (Study 3; n = 346) of ECCS. The collected ratings were highly consistent across the studies. Furthermore, we found that the level of climate change concern explained the intensity of elicited emotions. The ECCS dataset is available in Polish, Norwegian and English and can be employed for experimental research on climate communication, environmental attitudes, climate action-taking, or mental health and wellbeing.

A synergetic turn in cognitive neuroscience of brain diseases

Despite significant improvements in our understanding of brain diseases, many barriers remain. Cognitive neuroscience faces four major challenges: complex structure-function associations; disease phenotype heterogeneity; the lack of transdiagnostic models; and oversimplified cognitive approaches restricted to the laboratory. Here, we propose a synergetics framework that can help to perform the necessary dimensionality reduction of complex interactions between the brain, body, and environment. The key solutions include low-dimensional spatiotemporal hierarchies for brain-structure associations, whole-brain modeling to handle phenotype diversity, model integration of shared transdiagnostic pathophysiological pathways, and naturalistic frameworks balancing experimental control and ecological validity. Creating whole-brain models with reduced manifolds combined with ecological measures can improve our understanding of brain disease and help identify novel interventions. Synergetics provides an integrated framework for future progress in clinical and cognitive neuroscience, pushing the boundaries of brain health and disease toward more mature, naturalistic approaches.

Computational Language Modeling and the Promise of In Silico Experimentation

Language neuroscience currently relies on two major experimental paradigms: controlled experiments using carefully hand-designed stimuli, and natural stimulus experiments. These approaches have complementary advantages which allow them to address distinct aspects of the neurobiology of language, but each approach also comes with drawbacks. Here we discuss a third paradigm-in silico experimentation using deep learning-based encoding models-that has been enabled by recent advances in cognitive computational neuroscience. This paradigm promises to combine the interpretability of controlled experiments with the generalizability and broad scope of natural stimulus experiments. We show four examples of simulating language neuroscience experiments in silico and then discuss both the advantages and caveats of this approach.

How does the human brain process noisy speech in real life? Insights from the second-person neuroscience perspective

Comprehending speech with the existence of background noise is of great importance for human life. In the past decades, a large number of psychological, cognitive and neuroscientific research has explored the neurocognitive mechanisms of speech-in-noise comprehension. However, as limited by the low ecological validity of the speech stimuli and the experimental paradigm, as well as the inadequate attention on the high-order linguistic and extralinguistic processes, there remains much unknown about how the brain processes noisy speech in real-life scenarios. A recently emerging approach, i.e., the second-person neuroscience approach, provides a novel conceptual framework. It measures both of the speaker's and the listener's neural activities, and estimates the speaker-listener neural coupling with regarding of the speaker's production-related neural activity as a standardized reference. The second-person approach not only promotes the use of naturalistic speech but also allows for free communication between speaker and listener as in a close-to-life context. In this review, we first briefly review the previous discoveries about how the brain processes speech in noise; then, we introduce the principles and advantages of the second-person neuroscience approach and discuss its implications to unravel the linguistic and extralinguistic processes during speech-in-noise comprehension; finally, we conclude by proposing some critical issues and calls for more research interests in the second-person approach, which would further extend the present knowledge about how people comprehend speech in noise.

The neural basis of naturalistic semantic and social cognition

Decoding social environments and engaging meaningfully with other people are critical aspects of human cognition. Multiple cognitive systems, including social and semantic cognition, work alongside each other to support these processes. This study investigated shared processing between social and semantic systems using neuroimaging data collected during movie-viewing, which captures the multimodal environment in which social knowledge is exchanged. Semantic and social content from movie events (event-level) and movie transcripts (word-level) were used in parametric modulation analyses to test (1) the degree to which semantic and social information is processed within each respective network and (2) engagement of the same cross-network regions or the same domain-general hub located within the semantic network during semantic and social processing. Semantic word and event-level content engaged the same fronto-temporo-parietal network and a portion of the semantic hub in the anterior temporal lobe (ATL). Social word and event-level content engaged the supplementary motor area and right angular gyrus within the social network, but only social words engaged the domain-general semantic hub in left ATL. There was evidence of shared processing between the social and semantic systems in the dorsolateral portion of right ATL which was engaged by word and event-level semantic and social content. Overlap between the semantic and social word and event results was highly variable within and across participants, with the most consistent loci of overlap occurring in left inferior frontal, bilateral precentral and supramarginal gyri for social and semantic words and in bilateral superior temporal gyrus extending from ATL posteriorly into supramarginal gyri for social and semantic events. These results indicate a complex pattern of shared and distinct regions for social and semantic cognition during naturalistic processing. PROTOCOL REGISTRATION: The stage 1 protocol for this Registered Report was accepted in principle on October 11, 2022. The protocol, as accepted by the journal, can be found at: https://doi.org/10.17605/OSF.IO/ACWQY .

The dorsomedial prefrontal cortex prioritizes social learning during rest

Sociality is a defining feature of the human experience: We rely on others to ensure survival and cooperate in complex social networks to thrive. Are there brain mechanisms that help ensure we quickly learn about our social world to optimally navigate it? We tested whether portions of the brain's default network engage "by default" to quickly prioritize social learning during the memory consolidation process. To test this possibility, participants underwent functional MRI (fMRI) while viewing scenes from the documentary film, Samsara. This film shows footage of real people and places from around the world. We normed the footage to select scenes that differed along the dimension of sociality, while matched on valence, arousal, interestingness, and familiarity. During fMRI, participants watched the "social" and "nonsocial" scenes, completed a rest scan, and a surprise recognition memory test. Participants showed superior social (vs. nonsocial) memory performance, and the social memory advantage was associated with neural pattern reinstatement during rest in the dorsomedial prefrontal cortex (DMPFC), a key node of the default network. Moreover, it was during early rest that DMPFC social pattern reinstatement was greatest and predicted subsequent social memory performance most strongly, consistent with the "prioritization" account. Results simultaneously update 1) theories of memory consolidation, which have not addressed how social information may be prioritized in the learning process, and 2) understanding of default network function, which remains to be fully characterized. More broadly, the results underscore the inherent human drive to understand our vastly social world.

Impact of hypomanic personality traits on brain functional connectivity during a dynamic theory-of-mind task

Hypomanic personality traits are present in the general population and represent a risk factor for developing bipolar disorder. This personality style, notably its social component, is linked to difficulties in theory of mind (i.e., ability to infer mental states). Exploring the neural correlates of mental states' inference in individuals with these personality traits can provide meaningful insights into the development of bipolar disorder. The aim of the present study was therefore to investigate the potential impact of hypomanic traits on brain activation and task-based connectivity strength during a dynamic theory of mind task in a nonclinical population. A total of 52 nonclinical participants were recruited, and hypomanic traits were assessed with the Hypomanic Personality Scale. The severity of hypomanic traits was positively associated with right middle and inferior frontal gyri activations (in high vs. low inference in nonemotional condition and emotion vs. no emotion in high inference, respectively). It was also associated with stronger connectivity between the salience network (i.e., bilateral putamen and pallidum) and bilateral superior temporal gyri (high inference in nonemotional condition), and between cerebellar and temporal areas (high inference in emotional condition). These changes may either reflect adaptations or differential processing, and further studies are therefore mandatory.

Left-Hemisphere Cortical Language Regions Respond Equally to Observed Dialogue and Monologue

Much of the language we encounter in our everyday lives comes in the form of conversation, yet the majority of research on the neural basis of language comprehension has used input from only one speaker at a time. Twenty adults were scanned while passively observing audiovisual conversations using functional magnetic resonance imaging. In a block-design task, participants watched 20 s videos of puppets speaking either to another puppet (the dialogue condition) or directly to the viewer (the monologue condition), while the audio was either comprehensible (played forward) or incomprehensible (played backward). Individually functionally localized left-hemisphere language regions responded more to comprehensible than incomprehensible speech but did not respond differently to dialogue than monologue. In a second task, participants watched videos (1-3 min each) of two puppets conversing with each other, in which one puppet was comprehensible while the other's speech was reversed. All participants saw the same visual input but were randomly assigned which character's speech was comprehensible. In left-hemisphere cortical language regions, the time course of activity was correlated only among participants who heard the same character speaking comprehensibly, despite identical visual input across all participants. For comparison, some individually localized theory of mind regions and right-hemisphere homologues of language regions responded more to dialogue than monologue in the first task, and in the second task, activity in some regions was correlated across all participants regardless of which character was speaking comprehensibly. Together, these results suggest that canonical left-hemisphere cortical language regions are not sensitive to differences between observed dialogue and monologue.

Hierarchical organization of social action features along the lateral visual pathway

Recent theoretical work has argued that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. Like visual representations in the ventral and dorsal streams, representations in the lateral stream are thought to be hierarchically organized. However, no prior studies have comprehensively investigated the organization of naturalistic, social visual content in the lateral stream. To address this question, we curated a naturalistic stimulus set of 250 3-s videos of two people engaged in everyday actions. Each clip was richly annotated for its low-level visual features, mid-level scene and object properties, visual social primitives (including the distance between people and the extent to which they were facing), and high-level information about social interactions and affective content. Using a condition-rich fMRI experiment and a within-subject encoding model approach, we found that low-level visual features are represented in early visual cortex (EVC) and middle temporal (MT) area, mid-level visual social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Communicative interactions, in particular, explained unique variance in regions of the STS after accounting for variance explained by all other labeled features. Taken together, these results provide support for representation of increasingly abstract social visual content-consistent with hierarchical organization-along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of the lateral visual pathway.

A normative model of brain responses to social scenarios reflects the maturity of children and adolescents' social-emotional abilities

The rapid brain maturation in childhood and adolescence accompanies the development of socio-emotional functioning. However, it is unclear how the maturation of the neural activity drives the development of socio-emotional functioning and individual differences. This study aimed to reflect the age dependence of inter-individual differences in brain responses to socio-emotional scenarios and to develop naturalistic imaging indicators to assess the maturity of socio-emotional ability at the individual level. Using three independent naturalistic imaging datasets containing healthy participants (n = 111, 21 and 122), we found and validated that age-modulated inter-individual concordance of brain responses to socio-emotional movies in specific brain regions. The similarity of an individual's brain response to the average response of older participants was defined as response typicality, which predicted an individual's emotion regulation strategies in adolescence and theory of mind (ToM) in childhood. Its predictive power was not superseded by age, sex, cognitive performance or executive function. We further showed that the movie's valence and arousal ratings grounded the response typicality. The findings highlight that forming typical brain response patterns may be a neural phenotype underlying the maturation of socio-emotional ability. The proposed response typicality represents a neuroimaging approach to measure individuals' maturity of cognitive reappraisal and ToM.

Attachment Reminders Trigger Widespread Synchrony across Multiple Brains

Infant stimuli elicit widespread neural and behavioral response in human adults, and such massive allocation of resources attests to the evolutionary significance of the primary attachment. Here, we examined whether attachment reminders also trigger cross-brain concordance and generate greater neural uniformity, as indicated by intersubject correlation. Human mothers were imaged twice in oxytocin/placebo administration design, and stimuli included four ecological videos of a standard unfamiliar mother and infant: two infant/mother alone (Alone) and two mother-infant dyadic contexts (Social). Theory-driven analysis measured cross-brain synchrony in preregistered nodes of the parental caregiving network (PCN), which integrates subcortical structures underpinning mammalian mothering with cortical areas implicated in simulation, mentalization, and emotion regulation, and data-driven analysis assessed brain-wide concordance using whole-brain parcellation. Results demonstrated widespread cross-brain synchrony in both the PCN and across the neuroaxis, from primary sensory/somatosensory areas, through insular-cingulate regions, to temporal and prefrontal cortices. The Social context yielded significantly more cross-brain concordance, with PCNs striatum, parahippocampal gyrus, superior temporal sulcus, ACC, and PFC displaying cross-brain synchrony only to mother-infant social cues. Moment-by-moment fluctuations in mother-infant social synchrony, ranging from episodes of low synchrony to tightly coordinated positive bouts, were tracked online by cross-brain concordance in the preregistered ACC. Findings indicate that social attachment stimuli, representing evolutionary-salient universal cues that require no verbal narrative, trigger substantial interbrain concordance and suggest that the mother-infant bond, an icon standing at the heart of human civilization, may function to glue brains into a unified experience and bind humans into social groups.SIGNIFICANCE STATEMENT Infant stimuli elicit widespread neural response in human adults, attesting to their evolutionary significance, but do they also trigger cross-brain concordance and induce neural uniformity among perceivers? We measured cross-brain synchrony to ecological mother-infant videos. We used theory-driven analysis, measuring cross-brain concordance in the parenting network, and data-driven analysis, assessing brain-wide concordance using whole-brain parcellation. Attachment cues triggered widespread cross-brain concordance in both the parenting network and across the neuroaxis. Moment-by-moment fluctuations in behavioral synchrony were tracked online by cross-brain variability in ACC. Attachment reminders bind humans' brains into a unitary experience and stimuli characterized by social synchrony enhance neural similarity among participants, describing one mechanism by which attachment bonds provide the neural template for the consolidation of social groups.

Narrative simulation of social experiences in naturalistic context - A neurocinematic approach

Narratives may be regarded as simulations of everyday social situations. They are key to studying the human mind in socio-culturally determined contexts as they allow anchoring to the common ground of embodied and environmentally-engaged cognition. Here we review recent findings from naturalistic neuroscience on neural functions in conditions that mimic lifelike situations. We will focus particularly on neurocinematics, a research field that applies mediated narratives as stimuli for neuroimaging experiments. During the last two decades, this paradigm has contributed to an accumulation of insights about the neural underpinnings of behavior and sense-making in various narratively contextualized situations particularly pertaining to socio-emotional encounters. One of the key questions in neurocinematics is, how do intersubjectively synchronized brain activations relate to subjective experiences? Another question we address is how to bring natural contexts into experimental studies. Seeking to respond to both questions, we suggest neurocinematic studies to examine three manifestations of the same phenomenon side-by-side: subjective experiences of narrative situations, unfolding of narrative stimulus structure, and neural processes that co-constitute the experience. This approach facilitates identifying experientially meaningful activity patterns in the brain and points out what they may mean in relation to shared and communicable contents. Via rich-featured and temporally contextualized narrative stimuli, neurocinematics attempts to contribute to emerging holistic theories of neural dynamics and connectomics explaining typical and atypical interindividual variability.

Naturalistic auditory stimuli with fNIRS prefrontal cortex imaging: A potential paradigm for disorder of consciousness diagnostics (a study with healthy participants)

Disorder of consciousness (DOC) is a devastating condition due to brain damage. A patient in this condition is non-responsive, but nevertheless might be conscious at least at some level. Determining the conscious level of DOC patients is important for both medical and ethical reasons, but reliably achieving this has been a major challenge. Naturalistic stimuli in combination with neuroimaging have been proposed as a promising approach for DOC patient diagnosis. Capitalizing on and extending this proposal, the goal of the present study conducted with healthy participants was to develop a new paradigm with naturalistic auditory stimuli and functional near-infrared spectroscopy (fNIRS) - an approach that can be used at the bedside. Twenty-four healthy participants passively listened to 9 min of auditory story, scrambled auditory story, classical music, and scrambled classical music segments while their prefrontal cortex activity was recorded using fNIRS. We found much higher intersubject correlation (ISC) during story compared to scrambled story conditions both at the group level and in the majority of individual subjects, suggesting that fNIRS imaging of the prefrontal cortex might be a sensitive method to capture neural changes associated with narrative comprehension. In contrast, the ISC during the classical music segment did not differ reliably from scrambled classical music and was also much lower than the story condition. Our main result is that naturalistic auditory stories with fNIRS might be used in a clinical setup to identify high-level processing and potential consciousness in DOC patients.

Higher levels of narrativity lead to similar patterns of posterior EEG activity across individuals

Introduction

The focus of cognitive and psychological approaches to narrative has not so much been on the elucidation of important aspects of narrative, but rather on using narratives as tools for the investigation of higher order cognitive processes elicited by narratives (e.g., understanding, empathy, etc.). In this study, we work toward a scalar model of narrativity, which can provide testable criteria for selecting and classifying communication forms in their level of narrativity. We investigated whether being exposed to videos with different levels of narrativity modulates shared neural responses, measured by inter-subject correlation, and engagement levels.

Methods

Thirty-two participants watched video advertisements with high-level and low-level of narrativity while their neural responses were measured through electroencephalogram. Additionally, participants' engagement levels were calculated based on the composite of their self-reported attention and immersion scores.

Results

Results demonstrated that both calculated inter-subject correlation and engagement scores for high-level video ads were significantly higher than those for low-level, suggesting that narrativity levels modulate inter-subject correlation and engagement.

Discussion

We believe that these findings are a step toward the elucidation of the viewers' way of processing and understanding a given communication artifact as a function of the narrative qualities expressed by the level of narrativity.

Cortical gradients during naturalistic processing are hierarchical and modality-specific

Understanding cortical topographic organization and how it supports complex perceptual and cognitive processes is a fundamental question in neuroscience. Previous work has characterized functional gradients that demonstrate large-scale principles of cortical organization. How these gradients are modulated by rich ecological stimuli remains unknown. Here, we utilize naturalistic stimuli via movie-fMRI to assess macroscale functional organization. We identify principal movie gradients that delineate separate hierarchies anchored in sensorimotor, visual, and auditory/language areas. At the opposite/heteromodal end of these perception-to-cognition axes, we find a more central role for the frontoparietal network along with the default network. Even across different movie stimuli, movie gradients demonstrated good reliability, suggesting that these hierarchies reflect a brain state common across different naturalistic conditions. The relative position of brain areas within movie gradients showed stronger and more numerous correlations with cognitive behavioral scores compared to resting state gradients. Together, these findings provide an ecologically valid representation of the principles underlying cortical organization while the brain is active and engaged in multimodal, dynamic perceptual and cognitive processing.

Aberrant functional connectivity of the bed nucleus of the stria terminalis and its age dependence in children and adolescents with social anxiety disorder

BACKGROUND

Social anxiety disorder (SAD) is a prevalent and impairing mental disorder among children and adolescents. The bed nucleus of the stria terminalis (BNST) plays a critical role in anxiety disorders, including valence surveillance and hypervigilance for potential threats. However, the role of BNST and its related functional network in children and adolescents with SAD has not been fully investigated. This study examined the aberration of BNST's functional connectivity and its age dependence in adolescents with SAD.

METHODS

Using a sample of 75 SAD patients and 75 healthy controls (HCs) children aged 9-18 years old, we delineated the group-by-age interaction of BNST-seeded functional connectivity (FC) during resting state and movie-watching. The relationships between BNST-seeded FC and clinical scores were also examined.

RESULTS

During movie viewing, the FC between the right BNST and the left amygdala, bilateral posterior cingulate cortex (PCC), bilateral superior temporal cortex, and right pericalcarine cortex showed a diagnostic group-by-age interaction. Compared to HCs, SAD patients showed a significant enhancement of the above FC at younger ages. Meanwhile, they showed an age-dependent decrease in FC between the right BNST and left amygdala. Furthermore, for SAD patients, FC between the right BNST and left amygdala during movie viewing was positively correlated with separation anxiety scores.

CONCLUSIONS

The right BNST plays an essential role in the aberrant brain functioning in children and adolescents with SAD. The atypicality of BNST's FC has remarkable age dependence in SAD, suggesting an association of SAD with neurodevelopmental traits.

Neural responses to biological motion distinguish autistic and schizotypal traits

Difficulties in social interactions characterize both autism and schizophrenia and are correlated in the neurotypical population. It is unknown whether this represents a shared etiology or superficial phenotypic overlap. Both conditions exhibit atypical neural activity in response to the perception of social stimuli and decreased neural synchronization between individuals. This study investigated if neural activity and neural synchronization associated with biological motion perception are differentially associated with autistic and schizotypal traits in the neurotypical population. Participants viewed naturalistic social interactions while hemodynamic brain activity was measured with fMRI, which was modeled against a continuous measure of the extent of biological motion. General linear model analysis revealed that biological motion perception was associated with neural activity across the action observation network. However, intersubject phase synchronization analysis revealed neural activity to be synchronized between individuals in occipital and parietal areas but desynchronized in temporal and frontal regions. Autistic traits were associated with decreased neural activity (precuneus and middle cingulate gyrus), and schizotypal traits were associated with decreased neural synchronization (middle and inferior frontal gyri). Biological motion perception elicits divergent patterns of neural activity and synchronization, which dissociate autistic and schizotypal traits in the general population, suggesting that they originate from different neural mechanisms.

Functional near-infrared spectroscopy imaging of the prefrontal cortex during a naturalistic comedy movie

Naturalistic stimulation (i.e., movies and auditory narratives of some minutes' length) has been a powerful approach to bringing more real-life experiences into laboratory experiments. Data-driven, intersubject correlation (ISC) analysis permits examining to what extent activity in a specific brain region correlates across participants during exposure to a naturalistic stimulus, as well as testing whether neural activity correlates with behavioral measures. Notably, most of the previous research with naturalistic stimuli was conducted using functional fMRI (fMRI). Here, we tested whether a naturalistic approach and the ISC are feasible using functional near-infrared spectroscopy (fNIRS) - the imaging method particularly suited for populations of patients and children. Fifty-three healthy adult participants watched twice a 3-min segment of a Charlie Chaplin movie while we recorded the brain activity on the surface of their prefrontal cortex using fNIRS. In addition, an independent group of 18 participants used a continuous scoring procedure to rate the extent to which they felt that different parts of the movie fragment were funny. Our two findings were as follows. First, we found higher-than-zero ISC in fNIRS signals in the prefrontal cortex lobes, a result that was particularly high in the oxygenated channels during the first repetition of the movie. Second, we found a significant negative correlation between oxygenated brain signals and ratings of the movie's humorousness. In a series of control analyses we demonstrated that this latter correlation could not be explained by various non-humor-related movie sensory properties (e.g., auditory volume and image brightness). The key overall outcome of the present study is that fNIRS in combination with the naturalistic paradigms and the ISC might be a sensitive and powerful research method to explore cognitive processing. Our results also suggest a potential role of the prefrontal cortex in humor appreciation.

A tale of two connectivities: intra- and inter-subject functional connectivity jointly enable better prediction of social abilities

Naturalistic functional magnetic resonance imaging (fMRI) paradigms, such as movie viewing, are attracting increased attention, given their ability to mimic the real-world cognitive demands on attention and multimodal sensory integration. Moreover, naturalistic paradigms allow for characterizing brain network responses associated with dynamic social cognition in a model-free manner using inter-subject functional connectivity (ISFC). While intra-subject functional connectivity (FC) characterizes the individual’s brain functional architecture, ISFC characterizes the neural coupling driven by time-locked extrinsic dynamic stimuli across individuals. Here, we hypothesized that ISFC and FC provide distinct and complementary information about individual differences in social cognition. To test this hypothesis, we examined a public movie-viewing fMRI dataset with 32 healthy adults and 90 typically developing children. Building three partial least squares regression (PLS) models to predict social abilities using FC and/or ISFC, we compared predictive performance to determine whether combining two connectivity measures could improve the prediction accuracy of individuals’ social-cognitive abilities measured by a Theory of Mind (ToM) assessment. Our results indicated that the joint model (ISFC + FC) yielded the highest predictive accuracy and significantly predicted individuals’ social cognitive abilities (rho = 0.34, p < 0.001). We also confirmed that the improved accuracy was not due to the increased feature dimensionality. In conclusion, we demonstrated that intra-/inter-subject connectivity encodes unique information about social abilities, and a joint investigation could help us gain a more complete understanding of the complex processes supporting social cognition.

Social communication therapy for children at risk for neurodevelopmental difficulties: Protocol for a clinical trial

Exposure to adverse environments are risk factors for neurodevelopmental problems in childhood. Children exposed to such environments may benefit from interventions that target social communication abilities, since these are protective factors for healthy neurodevelopment. This randomized controlled trial will test the efficacy of Paediatric Autism Communication Therapy (PACT) in improving social communication development in young children at risk for neurodevelopmental difficulties living in poverty in Brazil. Participants will be 160 children aged 2-4 years with lower-than-average social communication abilities and their primary caregivers. Child-caregiver dyads will be recruited from public childhood education centers in impoverished urban regions of the city of São Paulo, Brazil. Lower-than-average social communication abilities will be defined by standard scores (≤84) on the socialization and/or communication domains of the Vineland Adaptive Behavior Scales. Child-caregiver dyads will be randomized to receive 12 sessions of the PACT intervention (n = 80) or 5 months of community support as usual plus psychoeducation (n = 80). The primary outcome (parent-child interaction) and secondary outcomes (parent-reported social communication abilities and neurophysiological activity during a live social interaction) will be measured pre- and postintervention. This study may lead to new interventions for vulnerable young children in Brazil and better understanding of the neural mechanisms of PACT.

Inferring Brain State Dynamics Underlying Naturalistic Stimuli Evoked Emotion Changes With dHA-HMM

The brain functional mechanisms underlying emotional changes have been primarily studied based on the traditional task design with discrete and simple stimuli. However, the brain state transitions when exposed to continuous and naturalistic stimuli with rich affection variations remain poorly understood. This study proposes a dynamic hyperalignment algorithm (dHA) to functionally align the inter-subject neural activity. The hidden Markov model (HMM) was used to study how the brain dynamics responds to emotion during long-time movie-viewing activity. The results showed that dHA significantly improved inter-subject consistency and allowed more consistent temporal HMM states across participants. Afterward, grouping the emotions in a clustering dendrogram revealed a hierarchical grouping of the HMM states. Further emotional sensitivity and specificity analyses of ordered states revealed the most significant differences in happiness and sadness. We then compared the activation map in HMM states during happiness and sadness and found significant differences in the whole brain, but strong activation was observed during both in the superior temporal gyrus, which is related to the early process of emotional prosody processing. A comparison of the inter-network functional connections indicates unique functional connections of the memory retrieval and cognitive network with the cerebellum network during happiness. Moreover, the persistent bilateral connections among salience, cognitive, and sensorimotor networks during sadness may reflect the interaction between high-level cognitive networks and low-level sensory networks. The main results were verified by the second session of the dataset. All these findings enrich our understanding of the brain states related to emotional variation during naturalistic stimuli.

Using child-friendly movie stimuli to study the development of face, place, and object regions from age 3 to 12 years

Scanning young children while they watch short, engaging, commercially-produced movies has emerged as a promising approach for increasing data retention and quality. Movie stimuli also evoke a richer variety of cognitive processes than traditional experiments, allowing the study of multiple aspects of brain development simultaneously. However, because these stimuli are uncontrolled, it is unclear how effectively distinct profiles of brain activity can be distinguished from the resulting data. Here we develop an approach for identifying multiple distinct subject-specific Regions of Interest (ssROIs) using fMRI data collected during movie-viewing. We focused on the test case of higher-level visual regions selective for faces, scenes, and objects. Adults (N = 13) were scanned while viewing a 5.6-min child-friendly movie, as well as a traditional localizer experiment with blocks of faces, scenes, and objects. We found that just 2.7 min of movie data could identify subject-specific face, scene, and object regions. While successful, movie-defined ssROIS still showed weaker domain selectivity than traditional ssROIs. Having validated our approach in adults, we then used the same methods on movie data collected from 3 to 12-year-old children (N = 122). Movie response timecourses in 3-year-old children's face, scene, and object regions were already significantly and specifically predicted by timecourses from the corresponding regions in adults. We also found evidence of continued developmental change, particularly in the face-selective posterior superior temporal sulcus. Taken together, our results reveal both early maturity and functional change in face, scene, and object regions, and more broadly highlight the promise of short, child-friendly movies for developmental cognitive neuroscience.

Interacting brains revisited: A cross-brain network neuroscience perspective

Elucidating the neural basis of social behavior is a long‐standing challenge in neuroscience. Such endeavors are driven by attempts to extend the isolated perspective on the human brain by considering interacting persons' brain activities, but a theoretical and computational framework for this purpose is still in its infancy. Here, we posit a comprehensive framework based on bipartite graphs for interbrain networks and address whether they provide meaningful insights into the neural underpinnings of social interactions. First, we show that the nodal density of such graphs exhibits nonrandom properties. While the current hyperscanning analyses mostly rely on global metrics, we encode the regions' roles via matrix decomposition to obtain an interpretable network representation yielding both global and local insights. With Bayesian modeling, we reveal how synchrony patterns seeded in specific brain regions contribute to global effects. Beyond inferential inquiries, we demonstrate that graph representations can be used to predict individual social characteristics, outperforming functional connectivity estimators for this purpose. In the future, this may provide a means of characterizing individual variations in social behavior or identifying biomarkers for social interaction and disorders.

Social-affective features drive human representations of observed actions

Humans observe actions performed by others in many different visual and social settings. What features do we extract and attend when we view such complex scenes, and how are they processed in the brain? To answer these questions, we curated two large-scale sets of naturalistic videos of everyday actions and estimated their perceived similarity in two behavioral experiments. We normed and quantified a large range of visual, action-related, and social-affective features across the stimulus sets. Using a cross-validated variance partitioning analysis, we found that social-affective features predicted similarity judgments better than, and independently of, visual and action features in both behavioral experiments. Next, we conducted an electroencephalography experiment, which revealed a sustained correlation between neural responses to videos and their behavioral similarity. Visual, action, and social-affective features predicted neural patterns at early, intermediate, and late stages, respectively, during this behaviorally relevant time window. Together, these findings show that social-affective features are important for perceiving naturalistic actions and are extracted at the final stage of a temporal gradient in the brain.

The Cerebellar Involvement in Autism Spectrum Disorders: From the Social Brain to Mouse Models

Autism spectrum disorders (ASD) are pervasive neurodevelopmental disorders that include a variety of forms and clinical phenotypes. This heterogeneity complicates the clinical and experimental approaches to ASD etiology and pathophysiology. To date, a unifying theory of these diseases is still missing. Nevertheless, the intense work of researchers and clinicians in the last decades has identified some ASD hallmarks and the primary brain areas involved. Not surprisingly, the areas that are part of the so-called “social brain”, and those strictly connected to them, were found to be crucial, such as the prefrontal cortex, amygdala, hippocampus, limbic system, and dopaminergic pathways. With the recent acknowledgment of the cerebellar contribution to cognitive functions and the social brain, its involvement in ASD has become unmistakable, though its extent is still to be elucidated. In most cases, significant advances were made possible by recent technological developments in structural/functional assessment of the human brain and by using mouse models of ASD. Mouse models are an invaluable tool to get insights into the molecular and cellular counterparts of the disease, acting on the specific genetic background generating ASD-like phenotype. Given the multifaceted nature of ASD and related studies, it is often difficult to navigate the literature and limit the huge content to specific questions. This review fulfills the need for an organized, clear, and state-of-the-art perspective on cerebellar involvement in ASD, from its connections to the social brain areas (which are the primary sites of ASD impairments) to the use of monogenic mouse models.

BrainIAK: The Brain Imaging Analysis Kit

Functional magnetic resonance imaging (fMRI) offers a rich source of data for studying the neural basis of cognition. Here, we describe the Brain Imaging Analysis Kit (BrainIAK), an open-source, free Python package that provides computationally optimized solutions to key problems in advanced fMRI analysis. A variety of techniques are presently included in BrainIAK: intersubject correlation (ISC) and intersubject functional connectivity (ISFC), functional alignment via the shared response model (SRM), full correlation matrix analysis (FCMA), a Bayesian version of representational similarity analysis (BRSA), event segmentation using hidden Markov models, topographic factor analysis (TFA), inverted encoding models (IEMs), an fMRI data simulator that uses noise characteristics from real data (fmrisim), and some emerging methods. These techniques have been optimized to leverage the efficiencies of high-performance compute (HPC) clusters, and the same code can be seamlessly transferred from a laptop to a cluster. For each of the aforementioned techniques, we describe the data analysis problem that the technique is meant to solve and how it solves that problem; we also include an example Jupyter notebook for each technique and an annotated bibliography of papers that have used and/or described that technique. In addition to the sections describing various analysis techniques in BrainIAK, we have included sections describing the future applications of BrainIAK to real-time fMRI, tutorials that we have developed and shared online to facilitate learning the techniques in BrainIAK, computational innovations in BrainIAK, and how to contribute to BrainIAK. We hope that this manuscript helps readers to understand how BrainIAK might be useful in their research.

Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing

Recognizing others' social interactions is a crucial human ability. Using simple stimuli, previous studies have shown that social interactions are selectively processed in the superior temporal sulcus (STS), but prior work with movies has suggested that social interactions are processed in the medial prefrontal cortex (mPFC), part of the theory of mind network. It remains unknown to what extent social interaction selectivity is observed in real world stimuli when controlling for other covarying perceptual and social information, such as faces, voices, and theory of mind. The current study utilizes a functional magnetic resonance imaging (fMRI) movie paradigm and advanced machine learning methods to uncover the brain mechanisms uniquely underlying naturalistic social interaction perception. We analyzed two publicly available fMRI datasets, collected while both male and female human participants (n = 17 and 18) watched two different commercial movies in the MRI scanner. By performing voxel-wise encoding and variance partitioning analyses, we found that broad social-affective features predict neural responses in social brain regions, including the STS and mPFC. However, only the STS showed robust and unique selectivity specifically to social interactions, independent from other covarying features. This selectivity was observed across two separate fMRI datasets. These findings suggest that naturalistic social interaction perception recruits dedicated neural circuity in the STS, separate from the theory of mind network, and is a critical dimension of human social understanding.

Effects of social and emotional context on neural activation and synchrony during movie viewing

Sharing emotional experiences impacts how we perceive and interact with the world, but the neural mechanisms that support this sharing are not well characterized. In this study, participants (N = 52) watched videos in an MRI scanner in the presence of an unfamiliar peer. Videos varied in valence and social context (i.e., participants believed their partner was viewing the same (joint condition) or a different (solo condition) video). Reported togetherness increased during positive videos regardless of social condition, indicating that positive contexts may lessen the experience of being alone. Two analysis approaches were used to examine both sustained neural activity averaged over time and dynamic synchrony throughout the videos. Both approaches revealed clusters in the medial prefrontal cortex that were more responsive to the joint condition. We observed a time‐averaged social‐emotion interaction in the ventromedial prefrontal cortex, although this region did not demonstrate synchrony effects. Alternatively, social‐emotion interactions in the amygdala and superior temporal sulcus showed greater neural synchrony in the joint compared to solo conditions during positive videos, but the opposite pattern for negative videos. These findings suggest that positive stimuli may be more salient when experienced together, suggesting a mechanism for forming social bonds.

A systematic review of the effects of psychiatric medications on social cognition

INTRODUCTION

Social cognition is an important area of mental functioning relevant to psychiatric disorders and social functioning, that may be affected by psychiatric drug treatments. The aim of this review was to investigate the effects of medications with sedative properties, on social cognition.

METHOD

This systematic review included experimental and neuroimaging studies investigating drug effects on social cognition. Data quality was assessed using a modified Downs and Black checklist (Trac et al. CMAJ 188: E120-E129, 2016). The review used narrative synthesis to analyse the data.

RESULTS

40 papers were identified for inclusion, 11 papers investigating benzodiazepine effects, and 29 investigating antipsychotic effects, on social cognition. Narrative synthesis showed that diazepam impairs healthy volunteer's emotion recognition, with supporting neuroimaging studies showing benzodiazepines attenuate amygdala activity. Studies of antipsychotic effects on social cognition gave variable results. However, many of these studies were in patients already taking medication, and potential practice effects were identified due to short-term follow-ups.

CONCLUSION

Healthy volunteer studies suggest that diazepam reduces emotional processing ability. The effects of benzodiazepines on other aspects of social cognition, as well as the effects of antipsychotics, remain unclear. Interpretations of the papers in this review were limited by variability in measures, small sample sizes, and lack of randomisation. More robust studies are necessary to evaluate the impact of these medications on social cognition.

Simultaneous functional MRI of two awake marmosets

Social cognition is a dynamic process that requires the perception and integration of a complex set of idiosyncratic features between interacting conspecifics. Here we present a method for simultaneously measuring the whole-brain activation of two socially interacting marmoset monkeys using functional magnetic resonance imaging. MRI hardware (a radiofrequency coil and peripheral devices) and image-processing pipelines were developed to assess brain responses to socialization, both on an intra-brain and inter-brain level. Notably, the brain activation of a marmoset when viewing a second marmoset in-person versus when viewing a pre-recorded video of the same marmoset-i.e., when either capable or incapable of socially interacting with a visible conspecific-demonstrates increased activation in the face-patch network. This method enables a wide range of possibilities for potentially studying social function and dysfunction in a non-human primate model.

Emotion Recognition and Context in Adolescents with Autism Spectrum Disorder

Emotion recognition research in autism has provided conflicting results and has ignored the role of context. We examined if autistic adolescents use context to identify displayed and felt emotion. Twenty adolescents with autism and 20 age-matched neurotypical adolescents identified emotions from a standardised set of images. The groups also viewed videos scenes with actors displaying a feigned emotion masking their true feelings. Participants identified the displayed and felt emotions. Both groups identified emotions from static images equally well. In the video condition, the autism group was unable to distinguish between the displayed and felt emotions. Emotion research is often divorced from context. Our findings suggest that autistic individuals have difficulty integrating contextual cues when processing emotions.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Investigation of the neural correlates of mentalizing through the Dynamic Inference Task, a new naturalistic task of social cognition

Understanding others' intentions requires both the identification of social cues (e.g., emotional facial expressions, gaze direction) and the attribution of a mental state to another. The neural substrates of these processes have often been studied separately, and results are heterogeneous, in part attributable to the variety of paradigms used. The aim of the present study was to explore the neural regions underlying these sociocognitive processes, using a novel naturalistic task in which participants engage with human protagonists featured in videos. A total of 51 right-handed volunteers underwent functional magnetic resonance imaging while performing the Dynamic Inference Task (DIT), manipulating the degree of inference (high vs. low), the presence of emotion (emotional vs. nonemotional), and gaze direction (direct vs. averted). High nonemotional inference elicited neural activation in temporal regions encompassing the right posterior superior temporal sulcus. The presence (vs. absence) of emotion in the high-inference condition elicited a bilateral pattern of activation in internal temporal areas around the amygdala and orbitofrontal structures, as well as activation in the right dorsomedial part of the superior frontal gyrus and the left precuneus. On account of its dynamic, naturalistic approach, the DIT seems a suitable task for exploring social interactions and the way we interact with others, both in nonclinical and clinical populations.

Individualized video recommendation modulates functional connectivity between large scale networks

With the emergence of AI‐powered recommender systems and their extensive use in the video streaming service, questions and concerns also arise. Why can recommended video content continuously capture users' attention? What is the impact of long‐term exposure to personalized video content on one's behaviors and brain functions? To address these questions, we designed an fMRI experiment presenting participants with personally recommended videos and generally recommended ones. To examine how large‐scale networks were modulated by personalized video content, graph theory analysis was applied to investigate the interaction between seven networks, including the ventral and dorsal attention networks (VAN, DAN), frontal–parietal network (FPN), salience network (SN), and three subnetworks of default mode network (dorsal medial prefrontal (dMPFC), Core, and medial temporal lobe (MTL)). Our results showed that viewing nonpersonalized video content mainly enhanced the connectivity in the DAN‐FPN‐Core pathway, whereas viewing personalized ones increased not only the connectivity in this pathway but also the DAN‐VAN‐dMPFC pathway. In addition, both personalized and nonpersonalized short videos decreased the couplings between SN and VAN as well as between two DMN subsystems, Core and MTL. Collectively, these findings uncovered distinct patterns of network interactions in response to short videos and provided insights into potential neural mechanisms by which human behaviors are biased by personally recommended content.

Reorganization of the Social Brain in Individuals with Only One Intact Cerebral Hemisphere

Social cognition and emotion are ubiquitous human processes that recruit a reliable set of brain networks in healthy individuals. These brain networks typically comprise midline (e.g., medial prefrontal cortex) as well as lateral regions of the brain including homotopic regions in both hemispheres (e.g., left and right temporo-parietal junction). Yet the necessary roles of these networks, and the broader roles of the left and right cerebral hemispheres in socioemotional functioning, remains debated. Here, we investigated these questions in four rare adults whose right (three cases) or left (one case) cerebral hemisphere had been surgically removed (to a large extent) to treat epilepsy. We studied four closely matched healthy comparison participants, and also compared the patient findings to data from a previously published larger healthy comparison sample (n = 33). Participants completed standardized socioemotional and cognitive assessments to investigate social cognition. Functional magnetic resonance imaging (fMRI) data were obtained during passive viewing of a short, animated movie that distinctively recruits two social brain networks: one engaged when thinking about other agents’ internal mental states (e.g., beliefs, desires, emotions; so-called Theory of Mind or ToM network), and the second engaged when thinking about bodily states (e.g., pain, hunger; so-called PAIN network). Behavioral assessments demonstrated remarkably intact general cognitive functioning in all individuals with hemispherectomy. Social-emotional functioning was somewhat variable in the hemispherectomy participants, but strikingly, none of these individuals had consistently impaired social-emotional processing and none of the assessment scores were consistent with a psychiatric disorder. Using inter-region correlation analyses, we also found surprisingly typical ToM and PAIN networks, as well as typical differentiation of the two networks (in the intact hemisphere of patients with either right or left hemispherectomy), based on idiosyncratic reorganization of cortical activation. The findings argue that compensatory brain networks can process social and emotional information following hemispherectomy across different age levels (from 3 months to 20 years old), and suggest that social brain networks typically distributed across midline and lateral brain regions in this domain can be reorganized, to a substantial degree.

A Feature-Based Network Analysis and fMRI Meta-Analysis Reveal Three Distinct Types of Prosocial Decisions

Tasks that measure correlates of prosocial decision-making share one common feature: agents can make choices that increase the welfare of a beneficiary. However, prosocial decisions vary widely as a function of other task features. The diverse ways that prosociality is defined and the heterogeneity of prosocial decisions have created challenges for interpreting findings across studies and identifying their neural correlates. To overcome these challenges, we aimed to organize the prosocial decision-making task space of neuroimaging studies. We conducted a systematic search for studies in which participants made decisions to increase the welfare of others during functional magnetic resonance imaging. We identified shared and distinct features of these tasks and employed an unsupervised graph-based approach to assess how various forms of prosocial decision-making are related in terms of their low-level components (e.g. task features like potential cost to the agent or potential for reciprocity). Analyses uncovered three clusters of prosocial decisions, which we labeled as cooperation, equity and altruism. This feature-based representation of the task structure was supported by results of a neuroimaging meta-analysis that each type of prosocial decisions recruited diverging neural systems. Results clarify some of the existing heterogeneity in how prosociality is conceptualized and generate insight for future research and task paradigm development.

Cortical temporal hierarchy is immature in middle childhood

The development of successful social-cognitive abilities requires one to track, accumulate, and integrate knowledge of other people's mental states across time. Regions of the brain differ in their temporal scale (i.e., a cortical temporal hierarchy) and those receptive to long temporal windows may facilitate social-cognitive abilities; however, the cortical development of long timescale processing remains to be investigated. The current study utilized naturalistic viewing to examine cortical development of long timescale processing and its relation to social-cognitive abilities in middle childhood - a time of expanding social spheres and increasing social-cognitive abilities. We found that, compared to adults, children exhibited reduced low-frequency power in the temporo-parietal junction (TPJ) and reduced specialization for long timescale processing within the TPJ and other regions broadly implicated in the default mode network and higher-order visual processing. Further, specialization for long timescales within the right dorsal medial prefrontal cortex became more 'adult-like' as a function of children's comprehension of character mental states. These results suggest that cortical temporal hierarchy in middle childhood is immature and may be important for an accurate representation of complex naturalistic social stimuli during this age.