Temporal binding within and across events

Schemas, reinforcement learning and the medial prefrontal cortex

Schemas are rich and complex knowledge structures about the typical unfolding of events in a context; for example, a schema of a dinner at a restaurant. In this Perspective, we suggest that reinforcement learning (RL), a computational theory of learning the structure of the world and relevant goal-oriented behaviour, underlies schema learning. We synthesize literature about schemas and RL to offer that three RL principles might govern the learning of schemas: learning via prediction errors, constructing hierarchical knowledge using hierarchical RL, and dimensionality reduction through learning a simplified and abstract representation of the world. We then suggest that the orbitomedial prefrontal cortex is involved in both schemas and RL due to its involvement in dimensionality reduction and in guiding memory reactivation through interactions with posterior brain regions. Last, we hypothesize that the amount of dimensionality reduction might underlie gradients of involvement along the ventral-dorsal and posterior-anterior axes of the orbitomedial prefrontal cortex. More specific and detailed representations might engage the ventral and posterior parts, whereas abstraction might shift representations towards the dorsal and anterior parts of the medial prefrontal cortex.

Large language models can segment narrative events similarly to humans

Humans perceive discrete events such as "restaurant visits" and "train rides" in their continuous experience. One important prerequisite for studying human event perception is the ability of researchers to quantify when one event ends and another begins. Typically, this information is derived by aggregating behavioral annotations from several observers. Here, we present an alternative computational approach where event boundaries are derived using a large language model, GPT-3, instead of using human annotations. We demonstrate that GPT-3 can segment continuous narrative text into events. GPT-3-annotated events are significantly correlated with human event annotations. Furthermore, these GPT-derived annotations achieve a good approximation of the "consensus" solution (obtained by averaging across human annotations); the boundaries identified by GPT-3 are closer to the consensus, on average, than boundaries identified by individual human annotators. This finding suggests that GPT-3 provides a feasible solution for automated event annotations, and it demonstrates a further parallel between human cognition and prediction in large language models. In the future, GPT-3 may thereby help to elucidate the principles underlying human event perception.

Threat Impairs the Organization of Memory Around Motivational Context

Previous work highlighted a critical role for top-down goals in shifting memory organization, namely, through studying the downstream influences of event segmentation and task switching on free recall. Here, we extend these frameworks into the realm of motivation, by comparing how threat motivation influences memory organization by capturing free recall dynamics. In Study 1, we manipulated individuals' motivation to successfully encode information by the threat of exposure to aversive sounds for forgetting. In Study 2, we conducted a parallel study manipulating motivation via instruction rather than threat, allowing us to examine changes directly related to threat motivation. Our findings showed that motivation to avoid threat broadly enhances memory for items presented within a threatening context, regardless of whether items were directly associated with the threat or not. Concurrently, these memory enhancements coincide with a decrease in the organization of memory around motivationally relevant features. These results highlight the importance of considering motivational valence when conceptualizing memory organization within adaptive memory frameworks.

The Ubiquity of Time in Latent-cause Inference

Humans have an outstanding ability to generalize from past experiences, which requires parsing continuously experienced events into discrete, coherent units, and relating them to similar past experiences. Time is a key element in this process; however, how temporal information is used in generalization remains unclear. Latent-cause inference provides a Bayesian framework for clustering experiences, by building a world model in which related experiences are generated by a shared cause. Here, we examine how temporal information is used in latent-cause inference, using a novel task in which participants see "microbe" stimuli and explicitly report the latent cause ("strain") they infer for each microbe. We show that humans incorporate time in their inference of latent causes, such that recently inferred latent causes are more likely to be inferred again. In particular, a "persistent" model, in which the latent cause inferred for one observation has a fixed probability of continuing to cause the next observation, explains the data significantly better than two other time-sensitive models, although extensive individual differences exist. We show that our task and this model have good psychometric properties, highlighting their potential use for quantifying individual differences in computational psychiatry or in neuroimaging studies.

The Relationship between Event Boundary Strength and Pattern Shifts across the Cortical Hierarchy during Naturalistic Movie-viewing

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: Phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multivoxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-participant agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile in both group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual participant level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. In addition, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Do actions structure auditory memory? Action-based event segmentation effects on sensory responses, pupil dilation and sequential memory

Our actions shape our everyday experience: what we experience, how we perceive, and remember it are deeply affected by how we interact with the world. Performing an action to deliver a stimulus engages neurophysiological processes which are reflected in the modulation of sensory and pupil responses. We hypothesized that these processes shape memory encoding, parsing the experience by grouping self- and externally generated stimuli into differentiated events. Participants encoded sound sequences, in which either the first or last few sounds were self-generated and the rest externally generated. We tested recall of the sequential order of sounds that had originated from the same (within event) or different sources (across events). Memory performance was not higher for within-event sounds, suggesting that actions did not structure the memory representation. However, during encoding, we observed the expected electrophysiological response attenuation for self-generated sounds, together with increased pupil dilation triggered by actions. Moreover, at the boundary between events, physiological responses to the first sound from the new source were influenced by the direction of the source switch. Our results suggest that introducing actions creates a stronger contextual shift than removing them, even though actions do not directly contribute to memory performance. This study contributes to our understanding of how interacting with sensory input shapes experiences by exploring the relationships between action effects on sensory responses, pupil dilation, and memory encoding. Importantly, it challenges the notion of a meaningful contribution from low-level neurophysiological mechanisms associated with action execution in the modulation of the self-generation effect.

Building and Breaking the Chain: A Model of Reward Prediction Error Integration and Segmentation of Memory

Prediction errors drive reinforcement learning and organize episodic memory into distinct contexts, but do these effects interact? Here, we review the roles of midbrain dopamine, the locus coeruleus, and the hippocampus in event cognition to propose and simulate the theoretical influence of two prediction error signals in integrating versus segmenting events in memory. We suggest that signed reward prediction errors can build mental models of reward environments, increasing the contextual similarity (integration) of experiences with stronger, more stable reward expectations. On the other hand, unsigned reward prediction errors can signal a new model of the environment, generating a contextual shift (segmentation) between experiences that crossed them. We moreover predicted that these differences in contextual similarity give rise to distinct patterns of temporal-order memory. We combined these ideas in a computational model to account for a seemingly paradoxical pattern of temporal-order memory where greater representational distance helps order memory within context but impairs it across contexts. We found that simulating signed reward prediction error integration and unsigned reward prediction error segmentation differentially enabled the model to perform associative chaining, which involved reactivating items between two tested probes to assist with sequential retrieval. In summary, our simulations provide a unifying explanation for the varied ways that neuromodulatory systems may alter event cognition and memory.

Goal Shifts Structure Memories and Prioritize Event-defining Information in Memory

Every day, we encounter far more information than we could possibly remember. Thus, our memory systems must organize and prioritize the details from an experience that can adaptively guide the storage and retrieval of specific episodic events. Prior work has shown that shifts in internal goal states can function as event boundaries, chunking experiences into distinct and memorable episodes. In addition, at short delays, memory for contextual information at boundaries has been shown to be enhanced compared with items within each event. However, it remains unclear if these memory enhancements are limited to features that signal a meaningful transition between events. To determine how changes in dynamic goal states influence the organization and content of long-term memory, we designed a 2-day experiment in which participants viewed a series of black-and-white objects surrounded by a color border on a two-by-two grid. The location of the object on the grid determined which of two tasks participants performed on a given trial. To examine if distinct types of goal shifts modulate the effects of event segmentation, we changed the border color, the task, or both after every four items in a sequence. We found that goal shifts influenced temporal memory in a manner consistent with the formation of distinct events. However, for subjective memory representations in particular, these effects differed by the type of event boundary. Furthermore, to examine if goal shifts lead to the prioritization of goal-relevant features in longer lasting memories, we tested source memory for each object's color and grid location both immediately and after a 24-hr delay. On the immediate test, boundaries enhanced the memory for all concurrent source features compared with nonboundary items, but only if those boundaries involved a goal shift. In contrast, after a delay, the source memory was selectively enhanced for the feature relevant to the goal shift. These findings suggest that goals can adaptively structure memories by prioritizing contextual features that define a unique episode in memory.

Time after Time: Preserving Temporal Memories When Experiences Repeat

Remembering when events occur in time is fundamental to episodic memory. Yet, many experiences repeat over time creating the potential for interference when attempting to recall temporally specific memories. Here, we argue that temporal memories are protected, in part, by reinstatement of temporal context information that is triggered by stimulus repetitions. We motivate this argument by integrating seminal findings across several distinct literatures and methodologies. Specifically, we consider key insights from foundational behavioral studies of temporal memory, recent electrophysiological and neuroimaging approaches to measuring memory reinstatement, and computational models that describe how temporal context representations shape memory processes. We also note several open questions concerning how temporal context reinstatement might influence subsequent temporal memory, including potential mediating effects of event spacing and event boundaries. These ideas and questions have the potential to guide future research and, ultimately, to advance theoretical accounts of how we preserve temporal memories.

Causal and Chronological Relationships Predict Memory Organization for Nonlinear Narratives

While recounting an experience, one can employ multiple strategies to transition from one part to the next. For instance, if the event was learned out of linear order, one can recall events according to the time they were learned (temporal), similar events (semantic), events occurring nearby in time (chronological), or events produced by the current event (causal). To disentangle the importance of these factors, we had participants watch the nonlinear narrative, Memento, under different task instructions and presentation orders. For each scene of the film, we also separately computed semantic and causal networks. We then contrasted the evidence for temporal, semantic, chronological, or causal strategies during recall. Critically, there was stronger evidence for the causal and chronological strategies than semantic or temporal strategies. Moreover, the causal and chronological strategies outperformed the temporal one even when we asked participants to recall the film in the presented order, underscoring the fundamental nature of causal structure in scaffolding understanding and organizing recall. Nevertheless, time still marginally predicted recall transitions, suggesting it operates as a weak signal in the presence of more salient forms of structure. In addition, semantic and causal network properties predicted scene memorability, including a stronger role for incoming causes to an event than its outgoing effects. In summary, these findings highlight the importance of accounting for complex, causal networks in knowledge building and memory.

Recall as a Window into Hippocampally Defined Events

We experience the present as a continuous stream of information, but often experience the past in parcels of unique events or episodes. Decades of research have helped to articulate how we perform this event segmentation in the moment, as well as how events and their boundaries influence what we later remember. More recently, neuroscientific research has suggested that the hippocampus plays a role at critical moments during event formation alongside its established role in enabling subsequent recall. Here, we review and explore the relationship between event processing and recall with the perspective that it can be uniquely characterized by the contributions of the hippocampus and its interactions with the rest of the brain. Specifically, we highlight a growing number of empirical studies suggesting that the hippocampus is important for processing events that have just ended, bridging the gap between the prior and current event, and influencing the contents and trajectories of recalled information. We also catalogue and summarize the multifaceted sets of findings concerning how recall is influenced by event structure. Lastly, we discuss several exciting directions for future research and how our understanding of events might be enriched by characterizing them in terms of the operations of different regions of the brain.

Motifs of human hippocampal and cortical high frequency oscillations structure processing and memory of naturalistic stimuli

The discrete events of our narrative experience are organized by the neural substrate that underlies episodic memory. This narrative process is segmented into discrete units by event boundaries. This permits a replay process that acts to consolidate each event into a narrative memory. High frequency oscillations (HFOs) are a potential mechanism for synchronizing neural activity during these processes. Here, we use intracranial recordings from participants viewing and freely recalling a naturalistic stimulus. We show that hippocampal HFOs increase following event boundaries and that coincident hippocampal-cortical HFOs (co-HFOs) occur in cortical regions previously shown to underlie event segmentation (inferior parietal, precuneus, lateral occipital, inferior frontal cortices). We also show that event-specific patterns of co-HFOs that occur during event viewing re-occur following the subsequent three event boundaries (in decaying fashion) and also during recall. This is consistent with models that support replay as a mechanism for memory consolidation. Hence, HFOs may coordinate activity across brain regions serving widespread event segmentation, encode naturalistic memory, and bind representations to assemble memory of a coherent, continuous experience.

Event boundaries drive norepinephrine release and distinctive neural representations of space in the rodent hippocampus

Episodic memories are temporally segmented around event boundaries that tend to coincide with moments of environmental change. During these times, the state of the brain should change rapidly, or reset, to ensure that the information encountered before and after an event boundary is encoded in different neuronal populations. Norepinephrine (NE) is thought to facilitate this network reorganization. However, it is unknown whether event boundaries drive NE release in the hippocampus and, if so, how NE release relates to changes in hippocampal firing patterns. The advent of the new GRABNE sensor now allows for the measurement of NE binding with sub-second resolution. Using this tool in mice, we tested whether NE is released into the dorsal hippocampus during event boundaries defined by unexpected transitions between spatial contexts and presentations of novel objections. We found that NE binding dynamics were well explained by the time elapsed after each of these environmental changes, and were not related to conditioned behaviors, exploratory bouts of movement, or reward. Familiarity with a spatial context accelerated the rate in which phasic NE binding decayed to baseline. Knowing when NE is elevated, we tested how hippocampal coding of space differs during these moments. Immediately after context transitions we observed relatively unique patterns of neural spiking which settled into a modal state at a similar rate in which NE returned to baseline. These results are consistent with a model wherein NE release drives hippocampal representations away from a steady-state attractor. We hypothesize that the distinctive neural codes observed after each event boundary may facilitate long-term memory and contribute to the neural basis for the primacy effect.

Locus coeruleus activation 'resets' hippocampal event representations and separates adjacent memories

Memories reflect the ebb and flow of experiences, capturing unique and meaningful events from our lives. Using a combination of functional magnetic resonance imaging (fMRI), neuromelanin imaging, and pupillometry, we show that arousal and locus coeruleus (LC) activation transform otherwise continuous experiences into distinct episodic memories. As sequences unfold, encountering a context shift, or event boundary, triggers arousal and LC processes that predict later memory separation. Boundaries furthermore promote temporal pattern separation within left hippocampal dentate gyrus, which correlates with heightened LC responses to those same transition points. We also find that a neurochemical index of prolonged LC activation correlates with diminished arousal responses at boundaries, suggesting a connection between elevated LC output and impaired event processing. These findings align with the idea that arousal processes initiate a neural and memory 'reset' in response to significant changes, constructing the very episodes that define everyday memory.

Event representation at the scale of ordinary experience

Weeks are divided into weekdays and weekends; years into semesters and seasons; lives into stages like childhood, adulthood, and adolescence. How does the structure of experience shape memory? Though much work has examined event representation in human cognition, little work has explored event representation at the scale of ordinary experience. Here, we use shared experiences - in the form of popular television shows - to explore how memories are shaped by event structure at a large scale. We find that memories for events in these shows exhibit several hallmarks of event cognition. Namely, we find that memories are organized with respect to their event structure (boundaries), and that beginnings and endings are better remembered at multiple levels of the event hierarchy simultaneously. These patterns seem to be partially, but not fully, explained by the perceived story-relevance of events. Lastly, using a longitudinal design, we also show how event representations evolve over periods of several months. These results offer an understanding of event cognition at the scale of ordinary human lives.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organisation of memories remains unclear. Here, we designed a novel task - the Emotion Boundary Task - wherein participants viewed sequences of negative and neutral images surrounded by a colour border. We manipulated perceptual context (border colour), emotional-picture valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create events with a shared perceptual and/or emotional context. We measured memory for temporal order and temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distances were remembered as longer for images spanning neutral-to-negative shifts - suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently-presented negative (versus neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral-to-negative events produce temporal dilation in memory, with implications for adaptive emotional functioning.

The relationship between event boundary strength and pattern shifts across the cortical hierarchy during naturalistic movie-viewing

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multi-voxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-subject agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile both in group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual subject level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. Additionally, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas, and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Contextual familiarity rescues the cost of switching

Changes in context influence the way we form and structure memories. Yet, little is known about how qualitatively different types of context switches shape memory organization. The current experiments characterize how different features of context change influence the structure and organization of free recall. Participants completed a context switching paradigm in which we manipulated the rate of switches and prior experience with the contexts participants were switching between (repeated vs. novel). We measured free-recall performance and determined the extent to which participants organized items by the order in which they were encoded or the type of context with which they were originally presented. Across two experiments, we found and replicated that rapidly switching to novel, but not repeated contexts, impaired memory recall performance and biased memory towards a greater reliance on temporal information. Critically, we observed that these differences in performance may be due to distinctions in how participants organize their recalls when rapidly switching contexts. Results indicated that participants were less likely to only cluster their responses by the same context when the contexts were repeating at a high rate, as compared to when the contexts were novel. Overall, our findings support a model in which contextual familiarity rescues the costs associated with rapidly switching to new tasks or contexts.

Integrating and fragmenting memories under stress and alcohol

Stress can powerfully influence the way we form memories, particularly the extent to which they are integrated or situated within an underlying spatiotemporal and broader knowledge architecture. These different representations in turn have significant consequences for the way we use these memories to guide later behavior. Puzzlingly, although stress has historically been argued to promote fragmentation, leading to disjoint memory representations, more recent work suggests that stress can also facilitate memory binding and integration. Understanding the circumstances under which stress fosters integration will be key to resolving this discrepancy and unpacking the mechanisms by which stress can shape later behavior. Here, we examine memory integration at multiple levels: linking together the content of an individual experience, threading associations between related but distinct events, and binding an experience into a pre-existing schema or sense of causal structure. We discuss neural and cognitive mechanisms underlying each form of integration as well as findings regarding how stress, aversive learning, and negative affect can modulate each. In this analysis, we uncover that stress can indeed promote each level of integration. We also show how memory integration may apply to understanding effects of alcohol, highlighting extant clinical and preclinical findings and opportunities for further investigation. Finally, we consider the implications of integration and fragmentation for later memory-guided behavior, and the importance of understanding which type of memory representation is potentiated in order to design appropriate interventions.

The attentional boost effect in free recall dynamics

With the attentional boost effect (ABE), responding to a briefly presented target in a detection task enhances the encoding of other items presented at the same time. However, the effects of target detection on context memory for the event in which the stimulus appeared remain unclear. Here, we present findings from verbal free recall and recognition experiments that test the effects of target detection during encoding on temporal and relational aspects of context memory. Consistent with prior demonstrations of limited effects of target detection on context memory, in Experiment 1 there was no evidence that target detection influenced the likelihood of transitioning to items that were presented at similar times during encoding, or that were in the same encoding condition. These null effects were replicated in a second experiment, which added an old/new recognition and relational memory test. These results indicate that target detection during encoding has minimal effects on the formation of temporal associations between words in memory.

Toward an integrative account of internal and external determinants of event segmentation

Our daily experiences unfold continuously, but we remember them as a series of discrete events through a process called event segmentation. Prominent theories of event segmentation suggest that event boundaries in memory are triggered by significant shifts in the external environment, such as a change in one's physical surroundings. In this review, we argue for a fundamental extension of this research field to also encompass internal state changes as playing a key role in structuring event memory. Accordingly, we propose an expanded taxonomy of event boundary-triggering processes, and review behavioral and neuroscience research on internal state changes in three core domains: affective states, goal states, and motivational states. Finally, we evaluate how well current theoretical frameworks can accommodate the unique and interactive contributions of internal states to event memory. We conclude that a theoretical perspective on event memory that integrates both external environment and internal state changes allows for a more complete understanding of how the brain structures experiences, with important implications for future research in cognitive and clinical neuroscience.

Comparing episodic memory outcomes from walking augmented reality and stationary virtual reality encoding experiences

Episodic Memory (EM) is the neurocognitive capacity to consciously recollect personally experienced events in specific spatio-temporal contexts. Although the relevance of spatial and temporal information is widely acknowledged in the EM literature, it remains unclear whether and how EM performance and organisation is modulated by self-motion, and by motor- and visually- salient environmental features (EFs) of the encoding environment. This study examines whether and how EM is modulated by locomotion and the EFs encountered in a controlled lifelike learning route within a large-scale building. Twenty-eight healthy participants took part in a museum-tour encoding task implemented in walking Augmented Reality (AR) and stationary Virtual Reality (VR) conditions. EM performance and organisation were assessed immediately and 48-hours after trials using a Remember/Familiar recognition paradigm. Results showed a significant positive modulation effect of locomotion on distinctive EM aspects. Findings highlighted a significant performance enhancement effect of stairway-adjacent locations compared to dead-end and mid-route stimuli-presentation locations. The results of this study may serve as design criteria to facilitate neurocognitive rehabilitative interventions of EM. The underlying technological framework developed for this study represents a novel and ecologically sound method for evaluating EM processes in lifelike situations, allowing researchers a naturalistic perspective into the complex nature of EM.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organization of memories remains unclear. Here, we designed a novel task-the Emotion Boundary Task-wherein participants viewed sequences of negative and neutral images surrounded by a color border. We manipulated perceptual context (border color), emotional valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create encoding events comprised of image sequences with a shared perceptual and/or emotional context. We measured memory for temporal order and subjectively remembered temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distance was remembered as longer for images spanning neutral-to-negative shifts-suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of this negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently presented negative (compared to neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral to negative events produce temporal dilation in memory, which may be relevant for adaptive emotional functioning.

Event Integration and Temporal Differentiation: How Hierarchical Knowledge Emerges in Hippocampal Subfields through Learning

Everyday life is composed of events organized by changes in contexts, with each event containing an unfolding sequence of occurrences. A major challenge facing our memory systems is how to integrate sequential occurrences within events while also maintaining their details and avoiding over-integration across different contexts. We asked if and how distinct hippocampal subfields come to hierarchically and, in parallel, represent both event context and subevent occurrences with learning. Female and male human participants viewed sequential events defined as sequences of objects superimposed on shared color frames while undergoing high-resolution fMRI. Importantly, these events were repeated to induce learning. Event segmentation, as indexed by increased reaction times at event boundaries, was observed in all repetitions. Temporal memory decisions were quicker for items from the same event compared to across different events, indicating that events shaped memory. With learning, hippocampal CA3 multivoxel activation patterns clustered to reflect the event context, with more clustering correlated with behavioral facilitation during event transitions. In contrast, in the dentate gyrus (DG), temporally proximal items that belonged to the same event became associated with more differentiated neural patterns. A computational model explained these results by dynamic inhibition in the DG. Additional similarity measures support the notion that CA3 clustered representations reflect shared voxel populations, while DG's distinct item representations reflect different voxel populations. These findings suggest an interplay between temporal differentiation in the DG and attractor dynamics in CA3. They advance our understanding of how knowledge is structured through integration and separation across time and context.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

The effect of hippocampal subfield damage on rapid temporal integration through statistical learning and associative inference

INTRODUCTION

The hippocampus (HPC) supports integration of information across time, often indexed by associative inference (AI) and statistical learning (SL) tasks. In AI, an indirect association between stimuli that never appeared together is inferred, whereas SL involves learning item relationships by extracting regularities across experiences. A recent model of hippocampal function (Schapiro et al., 2017) proposes that the HPC can support temporal integration in both paradigms through its two distinct pathways.

METHODS

We tested this models' predictions in four patients with varying degrees of bilateral HPC damage and matched healthy controls, with two patients with complementary damage to either the monosynaptic or trisynaptic pathway. During AI, participants studied overlapping paired associates (AB, BC) and their memory was tested for premise pairs (AB) and for inferred pairs (AC). During SL, participants passively viewed a continuous picture sequence that contained an underlying structure of triplets that later had to be recognized.

RESULTS

Binomial distributions were used to calculate above chance performance at the individual level. For AI, patients with focal HPC damage were impaired at inference but could correctly infer pairs above chance once premise pair acquisition was equated to controls; however, the patient with HPC and cortical damage showed severe impairment at recalling premise and inferred pairs, regardless of accounting for premise pair performance. For SL, none of the patients performed above chance, but notably neither did most controls.

CONCLUSIONS

Associative inference of indirect relationships can be intact with HPC damage to either hippocampal pathways or the HPC more broadly, provided premise pairs can first be formed. Inference may remain intact through residual HPC tissue supporting premise pair acquisition, and/or through extra-hippocampal structures supporting inference at retrieval. Clear conclusions about hippocampal contributions to SL are precluded by low performance in controls, which we caution is not dissimilar to previous amnesic studies using the same task. This complicates interpretations of studies claiming necessity of hippocampal contributions to SL and warrants the use of a common and reliable task before conclusions can be drawn.

More than a moment: What does it mean to call something an 'event'?

Experiences are stored in the mind as discrete mental units, or 'events,' which influence-and are influenced by-attention, learning, and memory. In this way, the notion of an 'event' is foundational to cognitive science. However, despite tremendous progress in understanding the behavioral and neural signatures of events, there is no agreed-upon definition of an event. Here, we discuss different theoretical frameworks of event perception and memory, noting what they can and cannot account for in the literature. We then highlight key aspects of events that we believe should be accounted for in theories of event processing--in particular, we argue that the structure and substance of events should be better reflected in our theories and paradigms. Finally, we discuss empirical gaps in the event cognition literature and what the future of event cognition research may look like.

Time for Memories

The ability to store information about the past to dynamically predict and prepare for the future is among the most fundamental tasks the brain performs. To date, the problems of understanding how the brain stores and organizes information about the past (memory) and how the brain represents and processes temporal information for adaptive behavior have generally been studied as distinct cognitive functions. This Symposium explores the inherent link between memory and temporal cognition, as well as the potential shared neural mechanisms between them. We suggest that working memory and implicit timing are interconnected and may share overlapping neural mechanisms. Additionally, we explore how temporal structure is encoded in associative and episodic memory and, conversely, the influences of episodic memory on subsequent temporal anticipation and the perception of time. We suggest that neural sequences provide a general computational motif that contributes to timing and working memory, as well as the spatiotemporal coding and recall of episodes.

Visual event boundaries restrict anchoring effects in decision-making

Research on higher-level thought has revealed many principles of reasoning and decision-making but has rarely made contact with how we perceive the world in the first place. Here we show how a lower-level property of perception-the spontaneous and task-irrelevant segmentation of continuous visual stimulation into discrete events-can restrict one of the most notorious biases in decision-making: numerical anchoring. Subjects walked down a long room in an immersive three dimensional (3D) animation and then made a numerical judgment (e.g., of how much a suitcase is worth, or of how many hours of community service a minor crime deserved). Critically, some subjects passed through a doorway (a visual event boundary) during their virtual walk, while others did not-equating time, distance traveled, and visual complexity. The anchoring manipulation was especially innocuous, not appearing to be part of the experiment at all. Before the online trial began, subjects reported the two-digit numerical value from a visually distorted "CAPTCHA" ("to verify that you are human")-where this task-irrelevant anchor was either low (e.g., 29) or high (e.g., 92). With no doorway, we observed reliable anchoring effects: Higher CAPTCHA values produced higher estimates. With the doorway, however, such effects were attenuated or even eliminated. This generalized across tasks involving item valuations, factual questions, and legal judgments and in tests of both incidental and explicit anchoring. This demonstrates how spontaneous visual event segmentation can have profound consequences for higher-level thought.

Opposite effects of emotion and event segmentation on temporal order memory and object-context binding

Our daily lives unfold continuously, yet our memories are organised into distinct events, situated in a specific context of space and time, and chunked when this context changes (at event boundaries). Previous research showed that this process, termed event segmentation, enhances object-context binding but impairs temporal order memory. Physiologically, peaks in pupil dilation index event segmentation, similar to emotion-induced bursts of autonomic arousal. Emotional arousal also modulates object-context binding and temporal order memory. Yet, these two critical factors have not been systematically studied together. To address this gap, we ran a behavioural experiment using a paradigm validated to study event segmentation and extended it with emotion manipulation. During encoding, we sequentially presented greyscale objects embedded in coloured frames (colour changes defining events), with a neutral or aversive sound. During retrieval, we tested participants' memory of temporal order memory and object-colour binding. We found opposite effects of emotion and event segmentation on episodic memory. While event segmentation enhanced object-context binding, emotion impaired it. On the contrary, event segmentation impaired temporal order memory, but emotion enhanced it. These findings increase our understanding of episodic memory organisation in laboratory settings, and potentially in real life with perceptual changes and emotion fluctuations constantly interacting.

Dynamic emotional states shape the episodic structure of memory

Human emotions fluctuate over time. However, it is unclear how these shifting emotional states influence the organization of episodic memory. Here, we examine how emotion dynamics transform experiences into memorable events. Using custom musical pieces and a dynamic emotion-tracking tool to elicit and measure temporal fluctuations in felt valence and arousal, our results demonstrate that memory is organized around emotional states. While listening to music, fluctuations between different emotional valences bias temporal encoding process toward memory integration or separation. Whereas a large absolute or negative shift in valence helps segment memories into episodes, a positive emotional shift binds sequential representations together. Both discrete and dynamic shifts in music-evoked valence and arousal also enhance delayed item and temporal source memory for concurrent neutral items, signaling the beginning of new emotional events. These findings are in line with the idea that the rise and fall of emotions can sculpt unfolding experiences into memories of meaningful events.

Proximity to boundaries reveals spatial context representation in human hippocampal CA1

Recollection of real-world events is often accompanied by a sense of being in the place where the event transpired. Convergent evidence suggests the hippocampus plays a key role in supporting episodic memory by associating information with the time and place it was originally encountered. This representation is reinstated during memory retrieval. However, little is known about the roles of different subfields of the human hippocampus in this process. Research in humans and non-human animal models has suggested that spatial environmental boundaries have a powerful influence on spatial and episodic memory, as well as hippocampal representations of contexts and events. Here, we used high-resolution fMRI to investigate how boundaries influence hippocampal activity patterns during the recollection of objects encountered in different spatial contexts. During the encoding phase, participants viewed objects once in a naturalistic virtual reality task in which they passively explored two rooms in one of two houses. Following the encoding phase, participants were scanned while they recollected items in the absence of any spatial contextual information. Our behavioral results demonstrated that spatial context memory was enhanced for objects encountered near a boundary. Activity patterns in CA1 carried information about the spatial context associated with each of these boundary items. Exploratory analyses revealed that recollection performance was correlated with the fidelity of retrieved spatial context representations in anterior parahippocampal cortex and subiculum. Our results highlight the privileged role of boundaries in CA1 and suggest more generally a close relationship between memory for spatial contexts and representations in the hippocampus and parahippocampal region.

Multiple memory systems for efficient temporal order memory

We report distinct contributions of multiple memory systems to the retrieval of the temporal order of events. The neural dynamics related to the retrieval of movie scenes revealed that recalling the temporal order of close events elevates hippocampal theta power, like that observed for recalling close spatial relationships. In contrast, recalling far events increases beta power in the orbitofrontal cortex, reflecting recall based on the overall movie structure.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

Cognitive dependencies and psychological health correlates of coherence in autobiographical reasoning

Autobiographical reasoning is a process by which an individual creates a coherent life account. The degree of coherence in autobiographical reasoning has been related to psychological health correlates such as depression and self-esteem in previous studies, but with inconsistent findings. Similarly, the basic psychological processes required to achieve coherence have been previously studied with regard to intelligence, but infrequently, and also with mixed findings. In the present study, we first developed and evaluated a German version of the Awareness of Narrative Identity Questionnaire (ANIQ) as an established measure of self-reported coherence. Second, we tested for cognitive dependencies on intelligence and memory indices. Third, we analysed its associations with psychological health correlates. We assessed a sample of 272 participants and thereof 189 participants again two-weeks later. Results supported the assumptions of the German ANIQ's psychometric qualities (factor structure, test-retest reliability, invariance) and validity (with regard to self-consciousness, self-concept clarity, and written accounts of personal turning points). We found coherence to be independent of intelligence and verbal memory, but partially dependent on figural memory. Coherence was related to depression, positivity, self-esteem, and self-esteem stability, but not to anxiety, substantiating its salutogenic effects.

The multi-angle extended three-dimensional activities (META) stimulus set: A tool for studying event cognition

To study complex human activity and how it is perceived and remembered, it is valuable to have large-scale, well-characterized stimuli that are representative of such activity. We present the Multi-angle Extended Three-dimensional Activities (META) stimulus set, a structured and highly instrumented set of extended event sequences performed in naturalistic settings. Performances were captured with two color cameras and a Kinect v2 camera with color and depth sensors, allowing the extraction of three-dimensional skeletal joint positions. We tracked the positions and identities of objects for all chapters using a mixture of manual coding and an automated tracking pipeline, and hand-annotated the timings of high-level actions. We also performed an online experiment to collect normative event boundaries for all chapters at a coarse and fine grain of segmentation, which allowed us to quantify event durations and agreement across participants. We share these materials publicly to advance new discoveries in the study of complex naturalistic activity.

Target detection does not influence temporal memory

Target detection has been found to enhance memory for concurrently presented stimuli under dual-task conditions. This "attentional boost effect" is reminiscent of findings in the event memory literature, where conditions giving rise to event boundaries have been shown to enhance memory for boundary items. Target detection commonly requires a working memory update (e.g., adding to a covert mental target count), which is also thought to be a key contributor to creating event boundaries. However, whether target detection impacts temporal memory in similar ways as event boundaries remains unknown, because these two parallel literatures have used different types of memory tests, making direct comparisons difficult. In a preregistered experiment with sequential Bayes factor design, we examined whether target detection influences temporal binding between items by inserting target and nontarget stimuli during encoding of trial-unique object images, and then comparing subsequent temporal order and distance memory for image pairs that span a target or nontarget. We found that target detection enhanced recognition memory for target trial images but had no effect on temporal binding between items. In a follow-up experiment, we showed that when the encoding task required updating of task set rather than target count, event segmentation-related temporal memory effects were observed. These results document that target detection as such does not disrupt inter-item associations in memory, and that attention orienting in the absence of updating task sets does not create event boundaries. This suggests a key distinction between declarative and procedural working memory updates in segmenting events in memory.

Neural temporal context reinstatement of event structure during memory recall

The transformation of experiences into meaningful events and memories is intertwined with the notion of time. Temporal perception can influence, and be influenced by, segmenting continuous experience into meaningful events. Episodic memories formed from these events become associated with temporal information as well. However, it is less clear how temporal perception contributes to structuring events and organizing memory: whether it plays a more active or passive role, and whether this temporal information is encoded initially during perception or influenced by retrieval processes. To address these questions, we examined how event segmentation influences temporal representations during initial perception and memory retrieval, without testing temporal information explicitly. Using a neural measure of temporal context extracted from scalp electroencephalography in human participants (N = 170), we found reduced temporal context similarity between studied items separated by an event boundary when compared to items from the same event. Furthermore, while participants freely recalled list items, neural activity reflected reinstatement of temporal context representations from the study phase, including temporal disruption. A computational model of episodic memory, the context maintenance and retrieval (CMR) model, predicted these results, and made novel predictions regarding the influence of temporal disruption on recall order. These findings implicate the impact of event structure on memory organization via temporal representations, underscoring the role of temporal information in event segmentation and episodic memory. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Incidental temporal binding in rats: A novel behavioral task

We designed a behavioral task called One-Trial Trace Escape Reaction (OTTER), in which rats incidentally associate two temporally discontinuous stimuli: a neutral acoustic cue (CS) with an aversive stimulus (US) which occurs two seconds later (CS-2s-US sequence). Rats are first habituated to two similar environmental contexts (A and B), each consisting of an interconnected dark and light chamber. Next, rats experience the CS-2s-US sequence in the dark chamber of one of the contexts (either A or B); the US is terminated immediately after a rat escapes into the light chamber. The CS-2s-US sequence is presented only once to ensure the incidental acquisition of the association. The recall is tested 24 h later when rats are presented with only the CS in the alternate context (B or A), and their behavioral response is observed. Our results show that 59% of the rats responded to the CS by escaping to the light chamber, although they experienced only one CS-2s-US pairing. The OTTER task offers a flexible high throughput tool to study memory acquired incidentally after a single experience. Incidental one-trial acquisition of association between temporally discontinuous events may be one of the essential components of episodic memory formation.

Attentional fluctuations and the temporal organization of memory

Event boundaries and temporal context shape the organization of episodic memories. We hypothesized that attentional fluctuations during encoding serve as "events" that affect temporal context representations and recall organization. Individuals encoded trial-unique objects during a modified sustained attention task. Memory was tested with free recall. Response time variability during the encoding tasks was used to characterize "in the zone" and "out of the zone" attentional states. We predicted that: 1) "in the zone", vs. "out of the zone", attentional states should be more conducive to maintaining temporal context representations that can cue temporally organized recall; and 2) temporally distant "in the zone" states may enable more recall "leaps" across intervening items. We replicated several important findings in the sustained attention and memory fields, including more online errors during "out of the zone" vs. "in the zone" attentional states and recall that was temporally structured. Yet, across four studies, we found no evidence for either of our main hypotheses. Recall was robustly temporally organized, and there was no difference in recall organization for items encoded "in the zone" vs. "out of the zone". We conclude that temporal context serves as a strong scaffold for episodic memory, one that can support organized recall even for items encoded during relatively poor attentional states. We also highlight the numerous challenges in striking a balance between sustained attention tasks (long blocks of a repetitive task) and memory recall tasks (short lists of unique items) and describe strategies for researchers interested in uniting these two fields.

Evidence That Event Boundaries Are Access Points for Memory Retrieval

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

Multi-view manifold learning of human brain-state trajectories

The complexity of the human brain gives the illusion that brain activity is intrinsically high-dimensional. Nonlinear dimensionality-reduction methods such as uniform manifold approximation and t-distributed stochastic neighbor embedding have been used for high-throughput biomedical data. However, they have not been used extensively for brain activity data such as those from functional magnetic resonance imaging (fMRI), primarily due to their inability to maintain dynamic structure. Here we introduce a nonlinear manifold learning method for time-series data-including those from fMRI-called temporal potential of heat-diffusion for affinity-based transition embedding (T-PHATE). In addition to recovering a low-dimensional intrinsic manifold geometry from time-series data, T-PHATE exploits the data's autocorrelative structure to faithfully denoise and unveil dynamic trajectories. We empirically validate T-PHATE on three fMRI datasets, showing that it greatly improves data visualization, classification, and segmentation of the data relative to several other state-of-the-art dimensionality-reduction benchmarks. These improvements suggest many potential applications of T-PHATE to other high-dimensional datasets of temporally diffuse processes.

Mnemonic Content and Hippocampal Patterns Shape Judgments of Time

Our experience of time can feel dilated or compressed, rather than reflecting true "clock time." Although many contextual factors influence the subjective perception of time, it is unclear how memory accessibility plays a role in constructing our experience of and memory for time. Here, we used a combination of behavioral and functional MRI measures in healthy young adults (N = 147) to ask the question of how memory is incorporated into temporal duration judgments. Behaviorally, we found that event boundaries, which have been shown to disrupt ongoing memory integration processes, result in the temporal compression of duration judgments. Additionally, using a multivoxel pattern similarity analysis of functional MRI data, we found that greater temporal pattern change in the left hippocampus within individual trials was associated with longer duration judgments. Together, these data suggest that mnemonic processes play a role in constructing representations of time.

Event boundaries structure the contents of long-term memory in younger and older adults

Event boundaries impose structure on how events are stored in long-term memory. Research with young adults has shown that associations within events are stronger than those that cross event boundaries. Recently, this effect was observed in both young and old adults using movie stimuli (Davis, Chemnitz, et al., 2021). Here, we test whether this effect extends to written narratives. Young and old participants read a series of narratives that were interspersed with temporal shifts in the storyline meant to elicit the perception of an event boundary. Later, participants were cued with sentences and were asked to recall the sentence that immediately followed. We expected participants would have worse memory when a cue and correct answer flanked a boundary than when it did not. In Experiment 1, we found that despite older adults' lower performance overall, both age groups had lower accuracy for cues that flanked a boundary, compared to cues that elicited a response from within the same event. Experiment 2 replicated the results from Experiment 1. Our results support past work that did not find age differences in event perception and demonstrate that older and younger adults may store events similarly in long-term memory.

The tie that binds: temporal coding and adaptive emotion

Emotions are temporally dynamic, but the persistence of emotions outside of their appropriate temporal context is detrimental to health and well-being. Yet, precisely how temporal coding and emotional processing interact remains unclear. Recently unveiled temporal context representations in the hippocampus, entorhinal cortex (EC), and prefrontal cortex (PFC) support memory for what happened when. Here, we discuss how these neural temporal representations may interact with densely interconnected amygdala circuitry to shape emotional functioning. We propose a neuroanatomically informed framework suggesting that high-fidelity temporal representations linked to dynamic experiences promote emotion regulation and adaptive emotional memories. Then, we discuss how newly-identified synaptic and molecular features of amygdala-hippocampal projections suggest that intense, amygdala-dependent emotional responses may distort temporal-coding mechanisms. We conclude by identifying key avenues for future research.

Representational dynamics of memories for real-life events

The continuous flow of experience that characterizes real-life events is not recorded as such in episodic memory but is condensed as a succession of event segments separated by temporal discontinuities. To unravel the neural basis of this representational structure, we recorded real-life events using wearable camera technology and used fMRI to investigate brain activity during their temporal unfolding in memory. We found that, compared to the representation of static scenes in memory, dynamically unfolding memory representations were associated with greater activation of the posterior medial episodic network. Strikingly, by analyzing the autocorrelation of brain activity patterns at successive time points throughout the retrieval period, we found that this network showed higher temporal dynamics when recalling events that included a higher density of event segments. These results reveal the key role of the posterior medial network in representing the dynamic unfolding of the event segments that constitute real-world memories.

The neural bases for timing of durations

Durations are defined by a beginning and an end, and a major distinction is drawn between durations that start in the present and end in the future ('prospective timing') and durations that start in the past and end either in the past or the present ('retrospective timing'). Different psychological processes are thought to be engaged in each of these cases. The former is thought to engage a clock-like mechanism that accurately tracks the continuing passage of time, whereas the latter is thought to engage a reconstructive process that utilizes both temporal and non-temporal information from the memory of past events. We propose that, from a biological perspective, these two forms of duration 'estimation' are supported by computational processes that are both reliant on population state dynamics but are nevertheless distinct. Prospective timing is effectively carried out in a single step where the ongoing dynamics of population activity directly serve as the computation of duration, whereas retrospective timing is carried out in two steps: the initial generation of population state dynamics through the process of event segmentation and the subsequent computation of duration utilizing the memory of those dynamics.

Grounding the Attentional Boost Effect in Events and the Efficient Brain

Attention and memory for everyday experiences vary over time, wherein some moments are better attended and subsequently better remembered than others. These effects have been demonstrated in naturalistic viewing tasks with complex and relatively uncontrolled stimuli, as well as in more controlled laboratory tasks with simpler stimuli. For example, in the attentional boost effect (ABE), participants perform two tasks at once: memorizing a series of briefly presented stimuli (e.g., pictures of outdoor scenes) for a later memory test, and responding to other concurrently presented cues that meet pre-defined criteria (e.g., participants press a button for a blue target square and do nothing for a red distractor square). However, rather than increasing dual-task interference, attending to a target cue boosts, rather than impairs, subsequent memory for concurrently presented information. In this review we describe current data on the extent and limitations of the attentional boost effect and whether it may be related to activity in the locus coeruleus neuromodulatory system. We suggest that insight into the mechanisms that produce the attentional boost effect may be found in recent advances in the locus coeruleus literature and from understanding of how the neurocognitive system handles stability and change in everyday events. We consequently propose updates to an early account of the attentional boost effect, the dual-task interaction model, to better ground it in what is currently known about event cognition and the role that the LC plays in regulating brain states.