Leveraging the power of media to drive cognition: a media-informed approach to naturalistic neuroscience

Towards an ecologically valid naturalistic cognitive neuroscience of memory and event cognition

The landscape of human memory and event cognition research has witnessed a transformative journey toward the use of naturalistic contexts and tasks. In this review, we track this progression from abrupt, artificial stimuli used in extensively controlled laboratory experiments to more naturalistic tasks and stimuli that present a more faithful representation of the real world. We argue that in order to improve ecological validity, naturalistic study designs must consider the complexity of the cognitive phenomenon being studied. Then, we review the current state of "naturalistic" event segmentation studies and critically assess frequently employed movie stimuli. We evaluate recently developed tools like lifelogging and other extended reality technologies to help address the challenges we identified with existing naturalistic approaches. We conclude by offering some guidelines that can be used to design ecologically valid cognitive neuroscience studies of memory and event cognition.

Lifespan differences in hippocampal subregion connectivity patterns during movie watching

Age-related episodic memory decline is attributed to functional alternations in the hippocampus. Less clear is how aging affects the functional connections of the hippocampus to the rest of the brain during episodic memory processing. We examined fMRI data from the CamCAN dataset, in which a large cohort of participants watched a movie (N = 643; 18-88 years), a proxy for naturalistic episodic memory encoding. We examined connectivity profiles across the lifespan both within the hippocampus (anterior, posterior), and between the hippocampal subregions and cortical networks. Aging was associated with reductions in contralateral (left, right) but not ipsilateral (anterior, posterior) hippocampal subregion connectivity. Aging was primarily associated with increased coupling between the anterior hippocampus and regions affiliated with Control, Dorsal Attention and Default Mode networks, yet decreased coupling between the posterior hippocampus and a selection of these regions. Differences in age-related hippocampal-cortical, but not within-hippocampus circuitry selectively predicted worse memory performance. Our findings comprehensively characterize hippocampal functional topography in relation to cognition in older age, suggesting that shifts in cortico-hippocampal connectivity may be sensitive markers of age-related episodic memory decline.

Inter- and intra-subject similarity in network functional connectivity across a full narrative movie

Naturalistic paradigms, such as watching movies during functional magnetic resonance imaging, are thought to prompt the emotional and cognitive processes typically elicited in real life situations. Therefore, naturalistic viewing (NV) holds great potential for studying individual differences. Previous studies have primarily focused on using shorter movie clips, geared toward eliciting specific and often isolated emotions, while the potential behind using full narratives depicted in commercial movies as a proxy for real-life experiences has barely been explored. Here, we offer preliminary evidence that a full narrative movie (FNM), that is, a movie covering a complete narrative arc, can capture complex socio-affective dynamics and their links to individual differences. Using the studyforrest dataset, we investigated inter- and intra-subject similarity in network functional connectivity (NFC) of 14 meta-analytically defined networks across a full narrative, audio-visual movie split into eight consecutive movie segments. We characterized the movie segments by valence and arousal portrayed within the sequences, before utilizing a linear mixed model to analyze which factors explain inter- and intra-subject similarity. Our results show that the model best explaining inter-subject similarity comprised network, movie segment, valence and a movie segment by valence interaction. Intra-subject similarity was influenced significantly by the same factors and an additional three-way interaction between movie segment, valence and arousal. Overall, inter- and intra-subject similarity in NFC were sensitive to the ongoing narrative and emotions in the movie. We conclude that FNMs offer complex content and dynamics that might be particularly valuable for studying individual differences. Further characterization of movie features, such as the overarching narratives, that enhance individual differences is needed for advancing the potential of NV research.

Smartphone-based gaze estimation for in-home autism research

Atypical gaze patterns are a promising biomarker of autism spectrum disorder. To measure gaze accurately, however, it typically requires highly controlled studies in the laboratory using specialized equipment that is often expensive, thereby limiting the scalability of these approaches. Here we test whether a recently developed smartphone-based gaze estimation method could overcome such limitations and take advantage of the ubiquity of smartphones. As a proof-of-principle, we measured gaze while a small sample of well-assessed autistic participants and controls watched videos on a smartphone, both in the laboratory (with lab personnel) and in remote home settings (alone). We demonstrate that gaze data can be efficiently collected, in-home and longitudinally by participants themselves, with sufficiently high accuracy (gaze estimation error below 1° visual angle on average) for quantitative, feature-based analysis. Using this approach, we show that autistic individuals have reduced gaze time on human faces and longer gaze time on non-social features in the background, thereby reproducing established findings in autism using just smartphones and no additional hardware. Our approach provides a foundation for scaling future research with larger and more representative participant groups at vastly reduced cost, also enabling better inclusion of underserved communities.

The Prediction of Brain Activity from Connectivity: Advances and Applications

The human brain is composed of multiple, discrete, functionally specialized regions that are interconnected to form large-scale distributed networks. Using advanced brain-imaging methods and machine-learning analytical approaches, recent studies have demonstrated that regional brain activity during the performance of various cognitive tasks can be accurately predicted from patterns of task-independent brain connectivity. In this review article, we first present evidence for the predictability of brain activity from structural connectivity (i.e., white matter connections) and functional connectivity (i.e., temporally synchronized task-free activations). We then discuss the implications of such predictions to clinical populations, such as patients diagnosed with psychiatric disorders or neurologic diseases, and to the study of brain-behavior associations. We conclude that connectivity may serve as an infrastructure that dictates brain activity, and we pinpoint several open questions and directions for future research.

Inter- and intra-subject similarity in network functional connectivity across a full narrative movie

Naturalistic paradigms, such as watching movies during functional magnetic resonance imaging (fMRI), are thought to prompt the emotional and cognitive processes typically elicited in real life situations. Therefore, naturalistic viewing (NV) holds great potential for studying individual differences. However, in how far NV elicits similarity within and between subjects on a network level, particularly depending on emotions portrayed in movies, is currently unknown. We used the studyforrest dataset to investigate the inter- and intra-subject similarity in network functional connectivity (NFC) of 14 meta-analytically defined networks across a full narrative, audio-visual movie split into 8 consecutive movie segments. We characterized the movie segments by valence and arousal portrayed within the sequences, before utilizing a linear mixed model to analyze which factors explain inter- and intra-subject similarity. Our results showed that the model best explaining inter-subject similarity comprised network, movie segment, valence and a movie segment by valence interaction. Intra-subject similarity was influenced significantly by the same factors and an additional three-way interaction between movie segment, valence and arousal. Overall, inter- and intra-subject similarity in NFC were sensitive to the ongoing narrative and emotions in the movie. Lowest similarity both within and between subjects was seen in the emotional regulation network and networks associated with long-term memory processing, which might be explained by specific features and content of the movie. We conclude that detailed characterization of movie features is crucial for NV research.

Brain dynamics and spatiotemporal trajectories during threat processing

In the past decades, functional MRI research has investigated mental states and their brain bases in largely static fashion based on evoked responses during blocked and event-related designs. Despite some progress in naturalistic designs, our understanding of threat processing remains largely limited to those obtained with standard paradigms. In the present paper, we applied Switching Linear Dynamical Systems to uncover the dynamics of threat processing during a continuous threat-of-shock paradigm. Importantly, unlike studies in systems neuroscience that frequently assume that systems are decoupled from external inputs, we characterized both endogenous and exogenous contributions to dynamics. First, we demonstrated that the SLDS model learned the regularities of the experimental paradigm, such that states and state transitions estimated from fMRI time series data from 85 ROIs reflected both the proximity of the circles and their direction (approach vs. retreat). After establishing that the model captured key properties of threat-related processing, we characterized the dynamics of the states and their transitions. The results revealed that threat processing can profitably be viewed in terms of dynamic multivariate patterns whose trajectories are a combination of intrinsic and extrinsic factors that jointly determine how the brain temporally evolves during dynamic threat. We propose that viewing threat processing through the lens of dynamical systems offers important avenues to uncover properties of the dynamics of threat that are not unveiled with standard experimental designs and analyses.

Hierarchical organization of social action features along the lateral visual pathway

Recent theoretical work has argued that in addition to the classical ventral (what) and dorsal (where/how) visual streams, there is a third visual stream on the lateral surface of the brain specialized for processing social information. Like visual representations in the ventral and dorsal streams, representations in the lateral stream are thought to be hierarchically organized. However, no prior studies have comprehensively investigated the organization of naturalistic, social visual content in the lateral stream. To address this question, we curated a naturalistic stimulus set of 250 3-s videos of two people engaged in everyday actions. Each clip was richly annotated for its low-level visual features, mid-level scene and object properties, visual social primitives (including the distance between people and the extent to which they were facing), and high-level information about social interactions and affective content. Using a condition-rich fMRI experiment and a within-subject encoding model approach, we found that low-level visual features are represented in early visual cortex (EVC) and middle temporal (MT) area, mid-level visual social features in extrastriate body area (EBA) and lateral occipital complex (LOC), and high-level social interaction information along the superior temporal sulcus (STS). Communicative interactions, in particular, explained unique variance in regions of the STS after accounting for variance explained by all other labeled features. Taken together, these results provide support for representation of increasingly abstract social visual content-consistent with hierarchical organization-along the lateral visual stream and suggest that recognizing communicative actions may be a key computational goal of the lateral visual pathway.

Arousal modulates the amygdala-insula reciprocal connectivity during naturalistic emotional movie watching

Emotional arousal is a complex state recruiting distributed cortical and subcortical structures, in which the amygdala and insula play an important role. Although previous neuroimaging studies have showed that the amygdala and insula manifest reciprocal connectivity, the effective connectivities and modulatory patterns on the amygdala-insula interactions underpinning arousal are still largely unknown. One of the reasons may be attributed to static and discrete laboratory brain imaging paradigms used in most existing studies. In this study, by integrating naturalistic-paradigm (i.e., movie watching) functional magnetic resonance imaging (fMRI) with a computational affective model that predicts dynamic arousal for the movie stimuli, we investigated the effective amygdala-insula interactions and the modulatory effect of the input arousal on the effective connections. Specifically, the predicted dynamic arousal of the movie served as regressors in general linear model (GLM) analysis and brain activations were identified accordingly. The regions of interest (i.e., the bilateral amygdala and insula) were localized according to the GLM activation map. The effective connectivity and modulatory effect were then inferred by using dynamic causal modeling (DCM). Our experimental results demonstrated that amygdala was the site of driving arousal input and arousal had a modulatory effect on the reciprocal connections between amygdala and insula. Our study provides novel evidence to the underlying neural mechanisms of arousal in a dynamical naturalistic setting.

A natural language fMRI dataset for voxelwise encoding models

Speech comprehension is a complex process that draws on humans' abilities to extract lexical information, parse syntax, and form semantic understanding. These sub-processes have traditionally been studied using separate neuroimaging experiments that attempt to isolate specific effects of interest. More recently it has become possible to study all stages of language comprehension in a single neuroimaging experiment using narrative natural language stimuli. The resulting data are richly varied at every level, enabling analyses that can probe everything from spectral representations to high-level representations of semantic meaning. We provide a dataset containing BOLD fMRI responses recorded while 8 participants each listened to 27 complete, natural, narrative stories (~6 hours). This dataset includes pre-processed and raw MRIs, as well as hand-constructed 3D cortical surfaces for each participant. To address the challenges of analyzing naturalistic data, this dataset is accompanied by a python library containing basic code for creating voxelwise encoding models. Altogether, this dataset provides a large and novel resource for understanding speech and language processing in the human brain.

The audience who knew too much: investigating the role of spontaneous theory of mind on the processing of dramatic irony scenes in film

As in real life, cinema viewers rely on spontaneous theory of mind (SToM) to interpret characters' mental states. Thus, analyzing cinematic structures offers a unique opportunity to examine ecologically valid sociocognitive processes. We conducted a proof-of-concept study (N = 42) to explore how SToM inferences impact film event comprehension in dramatic irony scenes, where knowledge divergence exists between the audience and characters. We hypothesized that spectators would focus more on characters' mental states in such false-belief inducing scenarios compared to scenarios without such disparity. We used six Harold Lloyd silent comedy clips in a narrative comprehension and spontaneous mental state attribution study with a between-subject (Knowledge Manipulation: Installation vs. Control) and within-subject (Phase: Context vs. Exploitation) comparisons. We provided critical information unknown to the characters only to the Installation group and withheld it from the Control group. By comparing differences in participants' descriptions of the clips during the Context phase (varying across groups) and Exploitation phase (same across groups), we evaluated viewers' processing of the same scenes based on their false- or true-belief representations. Our findings indicate that the Installation group used more cognitive mental state words during the Exploitation phase relative to the Context phase, suggesting that exposure to undisclosed critical information enhances the frequency of spontaneous epistemic state inferences and integration into event models of the exploitation. This research advances neurocinematics by highlighting spontaneous sociocognitive processes in event perception and comprehension and provides a novel dramatic irony film corpus and measures for future moment-to-moment SToM processing studies across cognitive-behavioral, physiological, and neural levels.

Dense Sampling Approaches for Psychiatry Research: Combining Scanners and Smartphones

Together, data from brain scanners and smartphones have sufficient coverage of biology, psychology, and environment to articulate between-person differences in the interplay within and across biological, psychological, and environmental systems thought to underlie psychopathology. An important next step is to develop frameworks that combine these two modalities in ways that leverage their coverage across layers of human experience to have maximum impact on our understanding and treatment of psychopathology. We review literature published in the last 3 years highlighting how scanners and smartphones have been combined to date, outline and discuss the strengths and weaknesses of existing approaches, and sketch a network science framework heretofore underrepresented in work combining scanners and smartphones that can push forward our understanding of health and disease.

Neural unscrambling of temporal information during a nonlinear narrative

Although we must experience our lives chronologically, storytellers often manipulate the order in which they relay events. How the brain processes temporal information while encoding a nonlinear narrative remains unclear. Here, we use functional magnetic resonance imaging during movie watching to investigate which brain regions are sensitive to information about time in a narrative and test whether the representation of temporal context across a narrative is more influenced by the order in which events are presented or their underlying chronological sequence. Results indicate that medial parietal regions are sensitive to cued jumps through time over and above other changes in context (i.e., location). Moreover, when processing non-chronological narrative information, the precuneus and posterior cingulate engage in on-the-fly temporal unscrambling to represent information chronologically. Specifically, days that are closer together in chronological time are represented more similarly regardless of when they are presented in the movie, and this representation is consistent across participants. Additional analyses reveal a strong spatial signature associated with higher magnitude jumps through time. These findings are consistent with prior theorizing on medial parietal regions as central to maintaining and updating narrative situation models, and suggest the priority of chronological information when encoding narrative events.

Toward naturalistic neuroscience: Mechanisms underlying the flattening of brain hierarchy in movie-watching compared to rest and task

Identifying the functional specialization of the brain has moved from using cognitive tasks and resting state to using ecological relevant, naturalistic movies. We leveraged a large-scale neuroimaging dataset to directly investigate the hierarchical reorganization of functional brain activity when watching naturalistic films compared to performing seven cognitive tasks and resting. A thermodynamics-inspired whole-brain model paradigm revealed the generative underlying mechanisms for changing the balance in causal interactions between brain regions in different conditions. Paradoxically, the hierarchy is flatter for movie-watching, and the level of nonreversibility is significantly smaller in comparison to both rest and tasks, where the latter in turn have the highest levels of hierarchy and nonreversibility. The underlying mechanisms were revealed by the model-based generative effective connectivity (GEC). Naturalistic films could therefore provide a fast and convenient way to measure important changes in GEC (integrating functional and anatomical connectivity) found in, for example, neuropsychiatric disorders. Overall, this study demonstrates the benefits of moving toward a more naturalistic neuroscience.

Early stressful experiences are associated with reduced neural responses to naturalistic emotional and social content in children

How do children's experiences relate to their naturalistic emotional and social processing? Because children can struggle with tasks in the scanner, we collected fMRI data while 4-to-11-year-olds watched a short film with positive and negative emotional events, and rich parent-child interactions (n = 70). We captured broad, normative stressful experiences by examining socioeconomic status (SES) and stressful life events, as well as children's more proximal experiences with their parents. For a sub-sample (n = 30), parenting behaviors were measured during a parent-child interaction, consisting of a picture book, a challenging puzzle, and free play with novel toys. We characterized positive parenting behaviors (e.g., warmth, praise) and negative parenting behaviors (e.g., harsh tone, physical control). We found that higher SES was related to greater activity in medial orbitofrontal cortex during parent-child interaction movie events. Negative parenting behaviors were associated with less activation of the ventral tegmental area and cerebellum during positive emotional events. In a region-of-interest analysis, we found that stressful life events and negative parenting behaviors were associated with less activation of the amygdala during positive emotional events. These exploratory results demonstrate the promise of using movie fMRI to study how early experiences may shape emotional, social, and motivational processes.