Geometric models reveal behavioural and neural signatures of transforming experiences into memories

Temporal asymmetries in inferring unobserved past and future events

Unlike temporally symmetric inferences about simple sequences, inferences about our own lives are asymmetric: we are better able to infer the past than the future, since we remember our past but not our future. Here we explore whether there are asymmetries in inferences about the unobserved pasts and futures of other people's lives. In two experiments (analyses of the replication experiment were pre-registered), our participants view segments of two character-driven television dramas and write out what they think happens just before or after each just-watched segment. Participants are better at inferring unseen past (versus future) events. This asymmetry is driven by participants' reliance on characters' conversational references in the narrative, which tend to favor the past. This tendency is also replicated in a large-scale analysis of conversational references in natural conversations. Our work reveals a temporal asymmetry in how observations of other people's behaviors can inform inferences about the past and future.

Reconstructing Spatiotemporal Trajectories of Visual Object Memories in the Human Brain

How the human brain reconstructs, step-by-step, the core elements of past experiences is still unclear. Here, we map the spatiotemporal trajectories along which visual object memories are reconstructed during associative recall. Specifically, we inquire whether retrieval reinstates feature representations in a copy-like but reversed direction with respect to the initial perceptual experience, or alternatively, this reconstruction involves format transformations and regions beyond initial perception. Participants from two cohorts studied new associations between verbs and randomly paired object images, and subsequently recalled the objects when presented with the corresponding verb cue. We first analyze multivariate fMRI patterns to map where in the brain high- and low-level object features can be decoded during perception and retrieval, showing that retrieval is dominated by conceptual features, represented in comparatively late visual and parietal areas. A separately acquired EEG dataset is then used to track the temporal evolution of the reactivated patterns using similarity-based EEG-fMRI fusion. This fusion suggests that memory reconstruction proceeds from anterior frontotemporal to posterior occipital and parietal regions, in line with a conceptual-to-perceptual gradient but only partly following the same trajectories as during perception. Specifically, a linear regression statistically confirms that the sequential activation of ventral visual stream regions is reversed between image perception and retrieval. The fusion analysis also suggests an information relay to frontoparietal areas late during retrieval. Together, the results shed light onto the temporal dynamics of memory recall and the transformations that the information undergoes between the initial experience and its later reconstruction from memory.

Trajectories through semantic spaces in schizophrenia and the relationship to ripple bursts

Human cognition is underpinned by structured internal representations that encode relationships between entities in the world (cognitive maps). Clinical features of schizophrenia-from thought disorder to delusions-are proposed to reflect disorganization in such conceptual representations. Schizophrenia is also linked to abnormalities in neural processes that support cognitive map representations, including hippocampal replay and high-frequency ripple oscillations. Here, we report a computational assay of semantically guided conceptual sampling and exploit this to test a hypothesis that people with schizophrenia (PScz) exhibit abnormalities in semantically guided cognition that relate to hippocampal replay and ripples. Fifty-two participants [26 PScz (13 unmedicated) and 26 age-, gender-, and intelligence quotient (IQ)-matched nonclinical controls] completed a category- and letter-verbal fluency task, followed by a magnetoencephalography (MEG) scan involving a separate sequence-learning task. We used a pretrained word embedding model of semantic similarity, coupled to a computational model of word selection, to quantify the degree to which each participant's verbal behavior was guided by semantic similarity. Using MEG, we indexed neural replay and ripple power in a post-task rest session. Across all participants, word selection was strongly influenced by semantic similarity. The strength of this influence showed sensitivity to task demands (category > letter fluency) and predicted performance. In line with our hypothesis, the influence of semantic similarity on behavior was reduced in schizophrenia relative to controls, predicted negative psychotic symptoms, and correlated with an MEG signature of hippocampal ripple power (but not replay). The findings bridge a gap between phenomenological and neurocomputational accounts of schizophrenia.

img2fmri: a python package for predicting group-level fMRI responses to visual stimuli using deep neural networks

Here we introduce a new python package, img2fmri, to predict group-level fMRI responses to individual images. This prediction model uses an artificial deep neural network (DNN), as DNNs have been successful at predicting cortical responses in the human visual cortex when trained on real world visual categorization tasks. To validate our model, we predict fMRI responses to images our model has not previously seen from a new dataset. We then show how our frame-by-frame prediction model can be extended to a continuous visual stimulus by predicting an fMRI response to Pixar Animation Studio's short film Partly Cloudy. In analyzing the timepoint-timepoint similarity of our predicted fMRI response around human-annotated event boundaries in the movie, we find that our model outperforms the baseline model in describing the dynamics of the real fMRI response around these event boundaries, particularly in the timepoints just before and at an event. These analyses suggest that in visual areas of the brain, at least some of the temporal dynamics we see in the brain's processing of continuous, naturalistic stimuli can be explained by dynamics in the stimulus itself, since they can be predicted from our frame-by-frame model. All code, analyses, tutorials, and installation instructions can be found at https://github.com/dpmlab/img2fmri.

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Post-encoding Reactivation Is Related to Learning of Episodes in Humans

Prior animal and human studies have shown that post-encoding reinstatement plays an important role in organizing the temporal sequence of unfolding episodes in memory. Here, we investigated whether post-encoding reinstatement serves to promote the encoding of "one-shot" episodic learning beyond the temporal structure in humans. In Experiment 1, participants encoded sequences of pictures depicting unique and meaningful episodic-like events. We used representational similarity analysis on scalp EEG recordings during encoding and found evidence of rapid picture-elicited EEG pattern reinstatement at episodic offset (around 500 msec post-episode). Memory reinstatement was not observed between successive elements within an episode, and the degree of memory reinstatement at episodic offset predicted later recall for that episode. In Experiment 2, participants encoded a shuffled version of the picture sequences from Experiment 1, rendering each episode meaningless to the participant but temporally structured as in Experiment 1, and we found no evidence of memory reinstatement at episodic offset. These results suggest that post-encoding memory reinstatement is akin to the rapid formation of unique and meaningful episodes that unfold over time.

Fitness tracking reveals task-specific associations between memory, mental health, and physical activity

Physical activity can benefit both physical and mental well-being. Different forms of exercise (e.g., aerobic versus anaerobic; running versus walking, swimming, or yoga; high-intensity interval training versus endurance workouts; etc.) impact physical fitness in different ways. For example, running may substantially impact leg and heart strength but only moderately impact arm strength. We hypothesized that the mental benefits of physical activity might be similarly differentiated. We focused specifically on how different intensities of physical activity might relate to different aspects of memory and mental health. To test our hypothesis, we collected (in aggregate) roughly a century’s worth of fitness data. We then asked participants to fill out surveys asking them to self-report on different aspects of their mental health. We also asked participants to engage in a battery of memory tasks that tested their short and long term episodic, semantic, and spatial memory performance. We found that participants with similar physical activity habits and fitness profiles tended to also exhibit similar mental health and task performance profiles. These effects were task-specific in that different physical activity patterns or fitness characteristics varied with different aspects of memory, on different tasks. Taken together, these findings provide foundational work for designing physical activity interventions that target specific components of cognitive performance and mental health by leveraging low-cost fitness tracking devices.

Predicting memory from the network structure of naturalistic events

When we remember events, we often do not only recall individual events, but also the connections between them. However, extant research has focused on how humans segment and remember discrete events from continuous input, with far less attention given to how the structure of connections between events impacts memory. Here we conduct a functional magnetic resonance imaging study in which participants watch and recall a series of realistic audiovisual narratives. By transforming narratives into networks of events, we demonstrate that more central events-those with stronger semantic or causal connections to other events-are better remembered. During encoding, central events evoke larger hippocampal event boundary responses associated with memory formation. During recall, high centrality is associated with stronger activation in cortical areas involved in episodic recollection, and more similar neural representations across individuals. Together, these results suggest that when humans encode and retrieve complex real-world experiences, the reliability and accessibility of memory representations is shaped by their location within a network of events.

Leveraging the power of media to drive cognition: a media-informed approach to naturalistic neuroscience

So-called 'naturalistic' stimuli have risen in popularity in cognitive, social and affective neuroscience over the last 15 years. However, a critical property of these stimuli is frequently overlooked: Media-like film, television, books and podcasts-are 'fundamentally not natural'. They are deliberately crafted products meant to elicit particular human thought, emotion and behavior. Here, we argue for a more informed approach to adopting media stimuli in experimental paradigms. We discuss the pitfalls of combining stimuli that are designed for research with those that are designed for other purposes (e.g. entertainment) under the umbrella term of 'naturalistic' and present strategies to improve rigor in the stimulus selection process. We assert that experiencing media should be considered a task akin to any other experimental task(s) and explain how this shift in perspective will compel more nuanced and generalizable research using these stimuli. Throughout, we offer theoretical and practical knowledge from multidisciplinary media research to raise the standard for the treatment of media stimuli in neuroscience research.

Neural signatures associated with temporal compression in the verbal retelling of past events

When we retell our past experiences, we aim to reproduce some version of the original events; this reproduced version is often temporally compressed relative to the original. However, it is currently unclear how this compression manifests in brain activity. One possibility is that a compressed retrieved memory manifests as a neural pattern which is more dissimilar to the original, relative to a more detailed or vivid memory. However, we argue that measuring raw dissimilarity alone is insufficient, as it confuses a variety of interesting and uninteresting changes. To address this problem, we examine brain pattern changes that are consistent across people. We show that temporal compression in individuals’ retelling of past events predicts systematic encoding-to-recall transformations in several higher associative regions. These findings elucidate how neural representations are not simply reactivated, but can also be transformed due to temporal compression during a universal form of human memory expression: verbal retelling.

Brain patterns measured while participants first watched a movie in the fMRI scanner, then recalled the movie’s key narrative features, demonstrate that temporal compression in individuals’ retelling of past events predicts encoding-to-recall transformations.

Retrieval context determines whether event boundaries impair or enhance temporal order memory

Meaningful changes in context create "event boundaries", segmenting continuous experience into distinct episodes in memory. A foundational finding in this literature is that event boundaries impair memory for the temporal order of stimuli spanning a boundary compared to equally spaced stimuli within an event. This seems surprising in light of intuitions about memory in everyday life, where the order of within-event experiences (did I have coffee before the first bite of bagel?) often seems more difficult to recall than the order of events per se (did I have breakfast or do the dishes first?). Here, we aimed to resolve this discrepancy by manipulating whether stimuli carried information about their encoding context during retrieval, as they often do in everyday life (e.g., bagel-breakfast). In Experiments 1 and 2, we show that stimuli inherently associated with a unique encoding context produce a "flipped" order memory effect, whereby temporal memory was superior for cross-boundary than within-event item pairs. In Experiments 3 and 4, we added context information at retrieval to a standard laboratory event memory protocol where stimuli were encoded in the presence of arbitrary context cues (colored frames). We found that whether temporal order memory for cross-boundary stimuli was enhanced or impaired relative to within-event items depended on whether the context was present or absent during the memory test. Taken together, we demonstrate that the effect of event boundaries on temporal memory is malleable, and determined by the availability of context information at retrieval.

Using Network Science to Understand the Aging Lexicon: Linking Individuals' Experience, Semantic Networks, and Cognitive Performance

People undergo many idiosyncratic experiences throughout their lives that may contribute to individual differences in the size and structure of their knowledge representations. Ultimately, these can have important implications for individuals' cognitive performance. We review evidence that suggests a relationship between individual experiences, the size and structure of semantic representations, as well as individual and age differences in cognitive performance. We conclude that the extent to which experience-dependent changes in semantic representations contribute to individual differences in cognitive aging remains unclear. To help fill this gap, we outline an empirical agenda that utilizes network analysis and involves the concurrent assessment of large-scale semantic networks and cognitive performance in younger and older adults. We present preliminary data to establish the feasibility and limitations of such empirical, network-analytical approaches.

The hippocampus constructs narrative memories across distant events

Life's events are scattered throughout time, yet we often recall different events in the context of an integrated narrative. Prior research suggests that the hippocampus, which supports memory for past events, can support the integration of overlapping associations or separate events in memory. However, the conditions that lead to hippocampus-dependent memory integration are unclear. We used functional brain imaging to test whether the opportunity to form a larger narrative (narrative coherence) drives hippocampal memory integration. During encoding of fictional stories, patterns of hippocampal activity, including activity at boundaries between events, were more similar between distant events that formed one coherent narrative, compared with overlapping events taken from unrelated narratives. One day later, the hippocampus preferentially supported detailed recall of coherent narrative events, through reinstatement of hippocampal activity patterns from encoding. These findings demonstrate a key function of the hippocampus: the integration of events into a narrative structure for memory.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Brain state kinematics and the trajectory of task performance improvement

Dimensionality reduction techniques offer a unique perspective on brain state dynamics, in which systems-level activity can be tracked through the engagement of a small number of component trajectories. Used in combination with neuroimaging data collected during the performance of cognitive tasks, these approaches can expose the otherwise latent dimensions upon which the brain reconfigures in order to facilitate cognitive performance. Here, we utilized Principal Component Analysis to transform parcellated BOLD timeseries from an fMRI dataset in which 70 human subjects performed an instruction based visuomotor learning task into orthogonal low-dimensional components. We then used Linear Discriminant Analysis to maximise the mean differences between the low-dimensional signatures of fast-and-slow reaction times and early-and-late learners, while also conserving variance present within these groups. The resultant basis set allowed us to describe meaningful differences between these groups and, importantly, to detail the patterns of brain activity which underpin these differences. Our results demonstrate non-linear interactions between three key brain activation maps with convergent trajectories observed at higher task repetitions consistent with optimization. Furthermore, we show subjects with the greatest reaction time improvements have delayed recruitment of left dorsal and lateral prefrontal cortex, as well as deactivation in parts of the occipital lobe and motor cortex, and that the slowest performers have weaker recruitment of somatosensory association cortex and left ventral visual stream, as well as weaker deactivation in the dorsal lateral prefrontal cortex. Overall our results highlight the utility of a kinematic description of brain states, whereby reformatting data into low-dimensional trajectories sensitive to the subtleties of a task can capture non-linear trends in a tractable manner and permit hypothesis generation at the level of brain states.

Neural signatures of attentional engagement during narratives and its consequences for event memory

As we comprehend narratives, our attentional engagement fluctuates over time. Despite theoretical conceptions of narrative engagement as emotion-laden attention, little empirical work has characterized the cognitive and neural processes that comprise subjective engagement in naturalistic contexts or its consequences for memory. Here, we relate fluctuations in narrative engagement to patterns of brain coactivation and test whether neural signatures of engagement predict subsequent memory. In behavioral studies, participants continuously rated how engaged they were as they watched a television episode or listened to a story. Self-reported engagement was synchronized across individuals and driven by the emotional content of the narratives. In functional MRI datasets collected as different individuals watched the same show or listened to the same story, engagement drove neural synchrony, such that default mode network activity was more synchronized across individuals during more engaging moments of the narratives. Furthermore, models based on time-varying functional brain connectivity predicted evolving states of engagement across participants and independent datasets. The functional connections that predicted engagement overlapped with a validated neuromarker of sustained attention and predicted recall of narrative events. Together, our findings characterize the neural signatures of attentional engagement in naturalistic contexts and elucidate relationships among narrative engagement, sustained attention, and event memory.