Flexible reuse of cortico-hippocampal representations during encoding and recall of naturalistic events

Towards an ecologically valid naturalistic cognitive neuroscience of memory and event cognition

The landscape of human memory and event cognition research has witnessed a transformative journey toward the use of naturalistic contexts and tasks. In this review, we track this progression from abrupt, artificial stimuli used in extensively controlled laboratory experiments to more naturalistic tasks and stimuli that present a more faithful representation of the real world. We argue that in order to improve ecological validity, naturalistic study designs must consider the complexity of the cognitive phenomenon being studied. Then, we review the current state of "naturalistic" event segmentation studies and critically assess frequently employed movie stimuli. We evaluate recently developed tools like lifelogging and other extended reality technologies to help address the challenges we identified with existing naturalistic approaches. We conclude by offering some guidelines that can be used to design ecologically valid cognitive neuroscience studies of memory and event cognition.

The causal structure and computational value of narratives

Many human behavioral and brain imaging studies have used narratively structured stimuli (e.g., written, audio, or audiovisual stories) to better emulate real-world experience in the laboratory. However, narratives are a special class of real-world experience, largely defined by their causal connections across time. Much contemporary neuroscience research does not consider this key property. We review behavioral and neuroscientific work that speaks to how causal structure shapes comprehension of and memory for narratives. We further draw connections between this work and reinforcement learning, highlighting how narratives help link causes to outcomes in complex environments. By incorporating the plausibility of causal connections between classes of actions and outcomes, reinforcement learning models may become more ecologically valid, while simultaneously elucidating the value of narratives.

Familiarity enhances functional connectivity between visual and nonvisual regions of the brain during natural viewing

We explored the neural correlates of familiarity with people and places using a naturalistic viewing paradigm. Neural responses were measured using functional magnetic resonance imaging, while participants viewed a movie taken from Game of Thrones. We compared inter-subject correlations and functional connectivity in participants who were either familiar or unfamiliar with the TV series. Higher inter-subject correlations were found between familiar participants in regions, beyond the visual brain, that are typically associated with the processing of semantic, episodic, and affective information. However, familiarity also increased functional connectivity between face and scene regions in the visual brain and the nonvisual regions of the familiarity network. To determine whether these regions play an important role in face recognition, we measured responses in participants with developmental prosopagnosia (DP). Consistent with a deficit in face recognition, the effect of familiarity was significantly attenuated across the familiarity network in DP. The effect of familiarity on functional connectivity between face regions and the familiarity network was also attenuated in DP. These results show that the neural response to familiarity involves an extended network of brain regions and that functional connectivity between visual and nonvisual regions of the brain plays an important role in the recognition of people and places during natural viewing.

FMRI correlates of autobiographical memory: Comparing silent retrieval with narrated retrieval

FMRI studies of autobiographical memory (AM) retrieval typically ask subjects to retrieve memories silently to avoid speech-related motion artifacts. Recently, some fMRI studies have started to use overt (spoken) retrieval to probe moment-to-moment retrieved content. However, the extent to which the overt retrieval method alters fMRI activations during retrieval is unknown. Here we examined this question by eliciting unrehearsed AMs during fMRI scanning either overtly or silently, in the same subjects, in different runs. Differences between retrieval modality (silent vs. narrated) included greater activation for silent retrieval in the anterior hippocampus, left angular gyrus, PCC, and superior PFC, and greater activation for narrated retrieval in speech production regions, posterior hippocampus, and the DLPFC. To probe temporal dynamics, we divided each retrieval period into an initial search phase and a later elaboration phase. The activations during the search and elaboration phases were broadly similar regardless of modality, and these activations were in line with previous fMRI studies of AM temporal dynamics employing silent retrieval. For both retrieval modalities, search activated the hippocampus, mPFC, ACC, and PCC, and elaboration activated the left DLPFC and middle temporal gyri. To examine content-specific reactivation during retrieval, the timecourse of narrated memory content was transcribed and modeled. We observed dynamic activation associated with object content in the lateral occipital complex, and activation associated with scene content in the retrosplenial cortex. The current findings show that both silent and narrated AMs activate a broadly similar memory network, with some key differences, and add to current knowledge regarding the content-specific dynamics of AM retrieval. However, these observed differences between retrieval modality suggest that studies using overt retrieval should carefully consider this method's possible effects on cognitive and neural processing.

A shared neural code for perceiving and remembering social interactions in the human superior temporal sulcus

Recognizing and remembering social information is a crucial cognitive skill. Neural patterns in the superior temporal sulcus (STS) support our ability to perceive others' social interactions. However, despite the prominence of social interactions in memory, the neural basis of remembering social interactions is still unknown. To fill this gap, we investigated the brain mechanisms underlying memory of others' social interactions during free spoken recall of a naturalistic movie. By applying machine learning-based fMRI encoding analyses to densely labeled movie and recall data we found that a subset of the STS activity evoked by viewing social interactions predicted neural responses in not only held-out movie data, but also during memory recall. These results provide the first evidence that activity in the STS is reinstated in response to specific social content and that its reactivation underlies our ability to remember others' interactions. These findings further suggest that the STS contains representations of social interactions that are not only perceptually driven, but also more abstract or conceptual in nature.

Keeping track of who knows what in multiparty conversation despite severe memory impairment

Language use has long been understood to be tailored to the intended addressee, a process termed audience design. Audience design is reflected in multiple aspects of language use, including adjustments based on the addressee's knowledge about the topic at hand. In group settings, audience design depends on representations of multiple individuals, each of whom may have different knowledge about the conversational topic. A central question, then, concerns how these representations are encoded and retrieved in multiparty conversation where successful conversation requires keeping track of who knows what. In the present research, we probe the biological memory systems that are involved in this process of multiparty audience design. We present the results of two experiments that compare language use in persons with bilateral hippocampal damage and severe declarative memory impairment (amnesia), and demographically matched neurotypical comparison participants. Participants played a game in which they discussed abstract images with one partner in conversation, and then discussed the images again with the same partner or with a new partner in a three-party conversation. Neurotypical participants' language use reflected newly formed representations of which partner was familiar with which images. Participants with amnesia showed evidence of partner-specific audience design in multiparty conversation but it was attenuated, especially when success required rapid alternations between representations of common ground. The findings suggest partial independence of the formation and use of partner-specific representations from the hippocampal-dependent declarative memory system and highlight the unique contributions of the declarative memory system to flexible and dynamic language use.

Hippocampal-cortical interactions during event boundaries support retention of complex narrative events

According to most memory theories, encoding involves continuous communication between the hippocampus and neocortex, but recent work has shown that key moments at the end of an event, called event boundaries, may be especially critical for memory formation. We sought to determine how communication between the hippocampus and neocortical regions during the encoding of naturalistic events related to subsequent retrieval of those events and whether this was particularly important at event boundaries. Participants encoded and recalled two cartoon movies during fMRI scanning. Higher functional connectivity between the hippocampus and the posterior medial network (PMN) at an event's offset is related to the subsequent successful recall of that event. Furthermore, hippocampal-PMN offset connectivity also predicted the amount of detail retrieved after a 2-day delay. These data demonstrate that the relationship between memory encoding and hippocampal-neocortical interaction is dynamic and biased toward boundaries.

Cortico-hippocampal networks carry information about characters and their relationships in an extended narrative

Social information is a centerpiece of human experience. Despite a wealth of research into the way we understand social relationships and how aspects of social life might be supported by the brain, relatively little is known about how the brain represents individual people and their relationships with others. How do intrinsic networks in the brain track people and their connections in complex situations? Here, we sought to understand this issue using an open neuroimaging dataset in which people freely viewed "The Grand Budapest Hotel." Using support vector machine classification of fMRI activity patterns, we found that character identity could be decoded throughout subsystems of the brain's "Default Mode" Network, especially in regions of an Anterior Temporal and a Medial Prefrontal subsystem, as well as a Medial Temporal Network (MTN). We tested character relationships in two ways - onscreen co-occurrence and shared semantic information from an independent sample of character descriptions - and found evidence for these representations throughout the "Default Mode" Network, and the MTN. The extent to which each variant of character relationships fit neural patterns differed across networks, with abstract semantic relatedness being especially prominent in regions of Anterior Temporal and Medial Prefrontal Networks. These data show that subsystems of the brain's "Default Mode" Network and MTN carry information about individual people as well as their connections, and highlight a particularly strong role for the Anterior Temporal network in representing this information.

Mega-scale movie-fields in the mouse visuo-hippocampal network

Natural visual experience involves a continuous series of related images while the subject is immobile. How does the cortico-hippocampal circuit process a visual episode? The hippocampus is crucial for episodic memory, but most rodent single unit studies require spatial exploration or active engagement. Hence, we investigated neural responses to a silent movie (Allen Brain Observatory) in head-fixed mice without any task or locomotion demands, or rewards. Surprisingly, a third (33%, 3379/10263) of hippocampal -dentate gyrus, CA3, CA1 and subiculum- neurons showed movie-selectivity, with elevated firing in specific movie sub-segments, termed movie-fields, similar to the vast majority of thalamo-cortical (LGN, V1, AM-PM) neurons (97%, 6554/6785). Movie-tuning remained intact in immobile or spontaneously running mice. Visual neurons had >5 movie-fields per cell, but only ~2 in hippocampus. The movie-field durations in all brain regions spanned an unprecedented 1000-fold range: from 0.02s to 20s, termed mega-scale coding. Yet, the total duration of all the movie-fields of a cell was comparable across neurons and brain regions. The hippocampal responses thus showed greater continuous-sequence encoding than visual areas, as evidenced by fewer and broader movie-fields than in visual areas. Consistently, repeated presentation of the movie images in a fixed, but scrambled sequence virtually abolished hippocampal but not visual-cortical selectivity. The preference for continuous, compared to scrambled sequence was eight-fold greater in hippocampal than visual areas, further supporting episodic-sequence encoding. Movies could thus provide a unified way to probe neural mechanisms of episodic information processing and memory, even in immobile subjects, across brain regions, and species.

Age differences in the neural correlates of recollection: transient versus sustained fMRI effects

Prior functional magnetic resonance imaging findings in young adults indicate that recollection-sensitive neural regions dissociate according to the time courses of their respective recollection effects. Here, we examined whether such dissociations are also evident in older adults. Young and older participants encoded a series of word-image pairs, judging which of the denoted objects was the smaller. At the test, participants judged whether each of a series of test words was old or new. If a word was old, the requirement was to recall the associated image and maintain it over a variable delay period. Older adults demonstrated significantly lower associative memory performance than young adults. Transient recollection effects were identified in the left hippocampus, medial prefrontal cortex, and posterior cingulate, while sustained effects were widespread across left lateral cortex and were also evident in the bilateral striatum. Except for those in the left insula, all effects were age-invariant. These findings suggest that both transient and sustained recollection effects are largely stable across much of the healthy adult life span.

Predictions transform memories: How expected versus unexpected events are integrated or separated in memory

Our brains constantly generate predictions about the environment based on prior knowledge. Many of the events we experience are consistent with these predictions, while others might be inconsistent with prior knowledge and thus violate our predictions. To guide future behavior, the memory system must be able to strengthen, transform, or add to existing knowledge based on the accuracy of our predictions. We synthesize recent evidence suggesting that when an event is consistent with our predictions, it leads to neural integration between related memories, which is associated with enhanced associative memory, as well as memory biases. Prediction errors, in turn, can promote both neural integration and separation, and lead to multiple mnemonic outcomes. We review these findings and how they interact with factors such as memory reactivation, prediction error strength, and task goals, to offer insight into what determines memory for events that violate our predictions. In doing so, this review brings together recent neural and behavioral research to advance our understanding of how predictions shape memory, and why.

Feedback signals in visual cortex during episodic and schematic memory retrieval and their potential implications for aphantasia

Recent findings indicate that visual feedback derived from episodic memory can be traced down to the earliest stages of visual processing, whereas feedback stemming from schema-related memories only reach intermediate levels in the visual processing hierarchy. In this opinion piece, we examine these differences in light of the 'what' and 'where' streams of visual perception. We build upon this new framework to propose that the memory deficits observed in aphantasics might be better understood as a difference in high-level feedback processing along the 'what' stream, rather than an episodic memory impairment.

Neural representation dynamics reveal computational principles of cognitive task learning

During cognitive task learning, neural representations must be rapidly constructed for novel task performance, then optimized for robust practiced task performance. How the geometry of neural representations changes to enable this transition from novel to practiced performance remains unknown. We hypothesized that practice involves a shift from compositional representations (task-general activity patterns that can be flexibly reused across tasks) to conjunctive representations (task-specific activity patterns specialized for the current task). Functional MRI during learning of multiple complex tasks substantiated this dynamic shift from compositional to conjunctive representations, which was associated with reduced cross-task interference (via pattern separation) and behavioral improvement. Further, we found that conjunctions originated in subcortex (hippocampus and cerebellum) and slowly spread to cortex, extending multiple memory systems theories to encompass task representation learning. The formation of conjunctive representations hence serves as a computational signature of learning, reflecting cortical-subcortical dynamics that optimize task representations in the human brain.

Effects of schema on the relationship between post-encoding brain connectivity and subsequent durable memory

Schemas can facilitate memory consolidation. Studies have suggested that interactions between the hippocampus and the ventromedial prefrontal cortex (vmPFC) are important for schema-related memory consolidation. However, in humans, how schema accelerates the consolidation of new information and relates to durable memory remains unclear. To address these knowledge gaps, we used a human analogue of the rodent spatial schema task and resting-state fMRI to investigate how post-encoding brain networks can predict long-term memory performance in different schema conditions. After participants were trained to obtain schema-consistent or schema-inconsistent object-location associations, they learned new object-location associations. The new associations were tested after the post-encoding rest in the scanner and 24 h later outside the scanner. The Bayesian multilevel modelling was applied to analyse the post-encoding brain networks. The results showed that during the post-encoding, stronger vmPFC- anterior hippocampal connectivity was associated with durable memory in the schema-consistent condition, whereas stronger object-selective lateral occipital cortex (LOC)-ventromedial prefrontal connectivity and weaker connectivity inside the default mode network were associated with durable memory in the schema inconsistent condition. In addition, stronger LOC-anterior hippocampal connectivity was associated with memory in both schema conditions. These results shed light on how schemas reconfigure early brain networks, especially the prefrontal-hippocampal and stimuli-relevant cortical networks and influence long-term memory performance.

The effects of age on neural correlates of recollection: transient versus sustained fMRI effects

Prior fMRI findings in young adults indicate that recollection-sensitive neural regions dissociate according to the time courses of their respective recollection effects. Here, we examined whether such dissociations are also evident in older adults. Young and older participants encoded a series of word-object image pairs, judging which of the denoted objects was the smaller. At test, participants first judged whether a test word was old or new. For items judged old, they were required to recall the associated image and hold it in mind across a variable delay period. A post-delay cue denoted which of three judgments should be made on the retrieved image. Older adults demonstrated significantly lower associative memory performance than young adults. Replicating prior findings, transient recollection effects were identified in the left hippocampus, medial prefrontal cortex and posterior cingulate, while sustained effects were widespread across left lateral cortex and were also evident in the bilateral striatum. With the exception of those in the left insula, all effects were age-invariant. These findings add to the evidence that recollection-related BOLD effects in different neural regions can be temporally dissociated. Additionally, the findings suggest that both transient and sustained recollection effects are largely stable across much of the healthy adult lifespan.