The Ubiquity of Time in Latent-cause Inference

Humans have an outstanding ability to generalize from past experiences, which requires parsing continuously experienced events into discrete, coherent units, and relating them to similar past experiences. Time is a key element in this process; however, how temporal information is used in generalization remains unclear. Latent-cause inference provides a Bayesian framework for clustering experiences, by building a world model in which related experiences are generated by a shared cause. Here, we examine how temporal information is used in latent-cause inference, using a novel task in which participants see "microbe" stimuli and explicitly report the latent cause ("strain") they infer for each microbe. We show that humans incorporate time in their inference of latent causes, such that recently inferred latent causes are more likely to be inferred again. In particular, a "persistent" model, in which the latent cause inferred for one observation has a fixed probability of continuing to cause the next observation, explains the data significantly better than two other time-sensitive models, although extensive individual differences exist. We show that our task and this model have good psychometric properties, highlighting their potential use for quantifying individual differences in computational psychiatry or in neuroimaging studies.

Threat Impairs the Organization of Memory Around Motivational Context

Previous work highlighted a critical role for top-down goals in shifting memory organization, namely, through studying the downstream influences of event segmentation and task switching on free recall. Here, we extend these frameworks into the realm of motivation, by comparing how threat motivation influences memory organization by capturing free recall dynamics. In Study 1, we manipulated individuals' motivation to successfully encode information by the threat of exposure to aversive sounds for forgetting. In Study 2, we conducted a parallel study manipulating motivation via instruction rather than threat, allowing us to examine changes directly related to threat motivation. Our findings showed that motivation to avoid threat broadly enhances memory for items presented within a threatening context, regardless of whether items were directly associated with the threat or not. Concurrently, these memory enhancements coincide with a decrease in the organization of memory around motivationally relevant features. These results highlight the importance of considering motivational valence when conceptualizing memory organization within adaptive memory frameworks.

Do actions structure auditory memory? Action-based event segmentation effects on sensory responses, pupil dilation and sequential memory

Our actions shape our everyday experience: what we experience, how we perceive, and remember it are deeply affected by how we interact with the world. Performing an action to deliver a stimulus engages neurophysiological processes which are reflected in the modulation of sensory and pupil responses. We hypothesized that these processes shape memory encoding, parsing the experience by grouping self- and externally generated stimuli into differentiated events. Participants encoded sound sequences, in which either the first or last few sounds were self-generated and the rest externally generated. We tested recall of the sequential order of sounds that had originated from the same (within event) or different sources (across events). Memory performance was not higher for within-event sounds, suggesting that actions did not structure the memory representation. However, during encoding, we observed the expected electrophysiological response attenuation for self-generated sounds, together with increased pupil dilation triggered by actions. Moreover, at the boundary between events, physiological responses to the first sound from the new source were influenced by the direction of the source switch. Our results suggest that introducing actions creates a stronger contextual shift than removing them, even though actions do not directly contribute to memory performance. This study contributes to our understanding of how interacting with sensory input shapes experiences by exploring the relationships between action effects on sensory responses, pupil dilation, and memory encoding. Importantly, it challenges the notion of a meaningful contribution from low-level neurophysiological mechanisms associated with action execution in the modulation of the self-generation effect.

Building and Breaking the Chain: A Model of Reward Prediction Error Integration and Segmentation of Memory

Prediction errors drive reinforcement learning and organize episodic memory into distinct contexts, but do these effects interact? Here, we review the roles of midbrain dopamine, the locus coeruleus, and the hippocampus in event cognition to propose and simulate the theoretical influence of two prediction error signals in integrating versus segmenting events in memory. We suggest that signed reward prediction errors can build mental models of reward environments, increasing the contextual similarity (integration) of experiences with stronger, more stable reward expectations. On the other hand, unsigned reward prediction errors can signal a new model of the environment, generating a contextual shift (segmentation) between experiences that crossed them. We moreover predicted that these differences in contextual similarity give rise to distinct patterns of temporal-order memory. We combined these ideas in a computational model to account for a seemingly paradoxical pattern of temporal-order memory where greater representational distance helps order memory within context but impairs it across contexts. We found that simulating signed reward prediction error integration and unsigned reward prediction error segmentation differentially enabled the model to perform associative chaining, which involved reactivating items between two tested probes to assist with sequential retrieval. In summary, our simulations provide a unifying explanation for the varied ways that neuromodulatory systems may alter event cognition and memory.

Goal Shifts Structure Memories and Prioritize Event-defining Information in Memory

Every day, we encounter far more information than we could possibly remember. Thus, our memory systems must organize and prioritize the details from an experience that can adaptively guide the storage and retrieval of specific episodic events. Prior work has shown that shifts in internal goal states can function as event boundaries, chunking experiences into distinct and memorable episodes. In addition, at short delays, memory for contextual information at boundaries has been shown to be enhanced compared with items within each event. However, it remains unclear if these memory enhancements are limited to features that signal a meaningful transition between events. To determine how changes in dynamic goal states influence the organization and content of long-term memory, we designed a 2-day experiment in which participants viewed a series of black-and-white objects surrounded by a color border on a two-by-two grid. The location of the object on the grid determined which of two tasks participants performed on a given trial. To examine if distinct types of goal shifts modulate the effects of event segmentation, we changed the border color, the task, or both after every four items in a sequence. We found that goal shifts influenced temporal memory in a manner consistent with the formation of distinct events. However, for subjective memory representations in particular, these effects differed by the type of event boundary. Furthermore, to examine if goal shifts lead to the prioritization of goal-relevant features in longer lasting memories, we tested source memory for each object's color and grid location both immediately and after a 24-hr delay. On the immediate test, boundaries enhanced the memory for all concurrent source features compared with nonboundary items, but only if those boundaries involved a goal shift. In contrast, after a delay, the source memory was selectively enhanced for the feature relevant to the goal shift. These findings suggest that goals can adaptively structure memories by prioritizing contextual features that define a unique episode in memory.

Rapid Learning of Temporal Dependencies at Multiple Timescales

Our environment contains temporal information unfolding simultaneously at multiple timescales. How do we learn and represent these dynamic and overlapping information streams? We investigated these processes in a statistical learning paradigm with simultaneous short and long timescale contingencies. Human participants (n = 96) played a game where they learned to quickly click on a target image when it appeared in one of nine locations, in eight different contexts. Across contexts, we manipulated the order of target locations: at a short timescale, the order of pairs of sequential locations in which the target appeared; at a longer timescale, the set of locations that appeared in the first versus the second half of the game. Participants periodically predicted the upcoming target location, and later performed similarity judgments comparing the games based on their order properties. Participants showed context-dependent sensitivity to order information at both short and long timescales, with evidence of stronger learning for short timescales. We modeled the learning paradigm using a gated recurrent network trained to make immediate predictions, which demonstrated multilevel learning timecourses and patterns of sensitivity to the similarity structure of the games that mirrored human participants. The model grouped games with matching rule structure and dissociated games based on low-level order information more so than high-level order information. The work shows how humans and models can rapidly and concurrently acquire order information at different timescales.

Causal and Chronological Relationships Predict Memory Organization for Nonlinear Narratives

While recounting an experience, one can employ multiple strategies to transition from one part to the next. For instance, if the event was learned out of linear order, one can recall events according to the time they were learned (temporal), similar events (semantic), events occurring nearby in time (chronological), or events produced by the current event (causal). To disentangle the importance of these factors, we had participants watch the nonlinear narrative, Memento, under different task instructions and presentation orders. For each scene of the film, we also separately computed semantic and causal networks. We then contrasted the evidence for temporal, semantic, chronological, or causal strategies during recall. Critically, there was stronger evidence for the causal and chronological strategies than semantic or temporal strategies. Moreover, the causal and chronological strategies outperformed the temporal one even when we asked participants to recall the film in the presented order, underscoring the fundamental nature of causal structure in scaffolding understanding and organizing recall. Nevertheless, time still marginally predicted recall transitions, suggesting it operates as a weak signal in the presence of more salient forms of structure. In addition, semantic and causal network properties predicted scene memorability, including a stronger role for incoming causes to an event than its outgoing effects. In summary, these findings highlight the importance of accounting for complex, causal networks in knowledge building and memory.

Recall as a Window into Hippocampally Defined Events

We experience the present as a continuous stream of information, but often experience the past in parcels of unique events or episodes. Decades of research have helped to articulate how we perform this event segmentation in the moment, as well as how events and their boundaries influence what we later remember. More recently, neuroscientific research has suggested that the hippocampus plays a role at critical moments during event formation alongside its established role in enabling subsequent recall. Here, we review and explore the relationship between event processing and recall with the perspective that it can be uniquely characterized by the contributions of the hippocampus and its interactions with the rest of the brain. Specifically, we highlight a growing number of empirical studies suggesting that the hippocampus is important for processing events that have just ended, bridging the gap between the prior and current event, and influencing the contents and trajectories of recalled information. We also catalogue and summarize the multifaceted sets of findings concerning how recall is influenced by event structure. Lastly, we discuss several exciting directions for future research and how our understanding of events might be enriched by characterizing them in terms of the operations of different regions of the brain.

"All the Stars Will Be Wells with a Rusty Pulley": Neural Processing of the Social and Pragmatic Content in a Narrative

Making sense of natural language and narratives requires building and manipulating a situation model by adding incoming information to the model and using the context stored in the model to comprehend subsequent details and events. Situation model maintenance is supported by the default mode network (DMN), but comprehension of the individual moments in the narrative relies on access to the conceptual store within the semantic system. The present study examined how these systems are engaged by different narrative content to investigate whether highly informative, or semantic, content is a particularly strong driver of semantic system activation compared with contextually driven content that requires using the situation model, which might instead engage DMN regions. The study further investigated which subregions of the graded semantic hub in the left anterior temporal lobe (ATL) were engaged by the type of narrative content. To do this, we quantified the semantic, pragmatic, social, ambiguous, and emotional content for each sentence in a complete narrative, the English translation of The Little Prince. Increased activation in the transmodal hub in the ventral ATL was only observed for high semantic (i.e., informative) relative to low semantic sentences. Activation in the dorsolateral and ventrolateral ATL subregions was observed for both high relative to low semantic and social content sentences, but the ventrolateral ATL effects were more extensive in the social condition. There was high correspondence between the social and pragmatic content results, particularly in the ventrolateral ATL. We argue that the ventrolateral ATL may be particularly engaged by internal, or endogenous, processing demands, aided by functional connections between the anterior middle temporal gyrus and the DMN. Pragmatic and social content may have driven endogenous processing given the pervasive and plot-progressing nature of this content in the narrative. We put forward a revised account of how the semantic system is engaged in naturalistic contexts, a critical step toward better understanding real-world semantic and social processing.

Infants infer and predict coherent event interactions: Modeling cognitive development

Mental representations of the environment in infants are sparse and grow richer during their development. Anticipatory eye fixation studies show that infants aged around 7 months start to predict the goal of an observed action, e.g., an object targeted by a reaching hand. Interestingly, goal-predictive gaze shifts occur at an earlier age when the hand subsequently manipulates an object and later when an action is performed by an inanimate actor, e.g., a mechanical claw. We introduce CAPRI2 (Cognitive Action PRediction and Inference in Infants), a computational model that explains this development from a functional, algorithmic perspective. It is based on the theory that infants learn object files and events as they develop a physical reasoning system. In particular, CAPRI2 learns a generative event-predictive model, which it uses to both interpret sensory information and infer goal-directed behavior. When observing object interactions, CAPRI2 (i) interprets the unfolding interactions in terms of event-segmented dynamics, (ii) maximizes the coherence of its event interpretations, updating its internal estimates and (iii) chooses gaze behavior to minimize expected uncertainty. As a result, CAPRI2 mimics the developmental pathway of infants' goal-predictive gaze behavior. Our modeling work suggests that the involved event-predictive representations, longer-term generative model learning, and shorter-term retrospective and active inference principles constitute fundamental building blocks for the effective development of goal-predictive capacities.

Top-down attention shifts behavioral and neural event boundaries in narratives with overlapping event scripts

Understanding and remembering the complex experiences of everyday life relies critically on prior schematic knowledge about how events in our world unfold over time. How does the brain construct event representations from a library of schematic scripts, and how does activating a specific script impact the way that events are segmented in time? We developed a novel set of 16 audio narratives, each of which combines one of four location-relevant event scripts (restaurant, airport, grocery store, and lecture hall) with one of four socially relevant event scripts (breakup, proposal, business deal, and meet cute), and presented them to participants in an fMRI study and a separate online study. Responses in the angular gyrus, parahippocampal gyrus, and subregions of the medial prefrontal cortex (mPFC) were driven by scripts related to both location and social information, showing that these regions can track schematic sequences from multiple domains. For some stories, participants were primed to attend to one of the two scripts by training them to listen for and remember specific script-relevant episodic details. Activating a location-related event script shifted the timing of subjective event boundaries to align with script-relevant changes in the narratives, and this behavioral shift was mirrored in the timing of neural responses, with mPFC event boundaries (identified using a hidden Markov model) aligning to location-relevant rather than socially relevant boundaries when participants were location primed. Our findings demonstrate that neural event dynamics are actively modulated by top-down goals and provide new insight into how narrative event representations are constructed through the activation of temporally structured prior knowledge.

Towards an ecologically valid naturalistic cognitive neuroscience of memory and event cognition

The landscape of human memory and event cognition research has witnessed a transformative journey toward the use of naturalistic contexts and tasks. In this review, we track this progression from abrupt, artificial stimuli used in extensively controlled laboratory experiments to more naturalistic tasks and stimuli that present a more faithful representation of the real world. We argue that in order to improve ecological validity, naturalistic study designs must consider the complexity of the cognitive phenomenon being studied. Then, we review the current state of "naturalistic" event segmentation studies and critically assess frequently employed movie stimuli. We evaluate recently developed tools like lifelogging and other extended reality technologies to help address the challenges we identified with existing naturalistic approaches. We conclude by offering some guidelines that can be used to design ecologically valid cognitive neuroscience studies of memory and event cognition.

Modeling human activity comprehension at human scale: prediction, segmentation, and categorization

Humans form sequences of event models-representations of the current situation-to predict how activity will unfold. Multiple mechanisms have been proposed for how the cognitive system determines when to segment the stream of behavior and switch from one active event model to another. Here, we constructed a computational model that learns knowledge about event classes (event schemas), by combining recurrent neural networks for short-term dynamics with Bayesian inference over event classes for event-to-event transitions. This architecture represents event schemas and uses them to construct a series of event models. This architecture was trained on one pass through 18 h of naturalistic human activities. Another 3.5 h of activities were used to test each variant for agreement with human segmentation and categorization. The architecture was able to learn to predict human activity, and it developed segmentation and categorization approaching human-like performance. We then compared two variants of this architecture designed to better emulate human event segmentation: one transitioned when the active event model produced high uncertainty in its prediction and the other transitioned when the active event model produced a large prediction error. The two variants learned to segment and categorize events, and the prediction uncertainty variant provided a somewhat closer match to human segmentation and categorization-despite being given no feedback about segmentation or categorization. These results suggest that event model transitioning based on prediction uncertainty or prediction error can reproduce two important features of human event comprehension.

Reconciling shared versus context-specific information in a neural network model of latent causes

It has been proposed that, when processing a stream of events, humans divide their experiences in terms of inferred latent causes (LCs) to support context-dependent learning. However, when shared structure is present across contexts, it is still unclear how the "splitting" of LCs and learning of shared structure can be simultaneously achieved. Here, we present the Latent Cause Network (LCNet), a neural network model of LC inference. Through learning, it naturally stores structure that is shared across tasks in the network weights. Additionally, it represents context-specific structure using a context module, controlled by a Bayesian nonparametric inference algorithm, which assigns a unique context vector for each inferred LC. Across three simulations, we found that LCNet could (1) extract shared structure across LCs in a function learning task while avoiding catastrophic interference, (2) capture human data on curriculum effects in schema learning, and (3) infer the underlying event structure when processing naturalistic videos of daily events. Overall, these results demonstrate a computationally feasible approach to reconciling shared structure and context-specific structure in a model of LCs that is scalable from laboratory experiment settings to naturalistic settings.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organisation of memories remains unclear. Here, we designed a novel task - the Emotion Boundary Task - wherein participants viewed sequences of negative and neutral images surrounded by a colour border. We manipulated perceptual context (border colour), emotional-picture valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create events with a shared perceptual and/or emotional context. We measured memory for temporal order and temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distances were remembered as longer for images spanning neutral-to-negative shifts - suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently-presented negative (versus neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral-to-negative events produce temporal dilation in memory, with implications for adaptive emotional functioning.

Events in context-The HED framework for the study of brain, experience and behavior

The brain is a complex dynamic system whose current state is inextricably coupled to awareness of past, current, and anticipated future threats and opportunities that continually affect awareness and behavioral goals and decisions. Brain activity is driven on multiple time scales by an ever-evolving flow of sensory, proprioceptive, and idiothetic experience. Neuroimaging experiments seek to isolate and focus on some aspect of these complex dynamics to better understand how human experience, cognition, behavior, and health are supported by brain activity. Here we consider an event-related data modeling approach that seeks to parse experience and behavior into a set of time-delimited events. We distinguish between event processes themselves, that unfold through time, and event markers that record the experiment timeline latencies of event onset, offset, and any other event phase transitions. Precise descriptions of experiment events (sensory, motor, or other) allow participant experience and behavior to be interpreted in the context either of the event itself or of all or any experiment events. We discuss how events in neuroimaging experiments have been, are currently, and should best be identified and represented with emphasis on the importance of modeling both events and event context for meaningful interpretation of relationships between brain dynamics, experience, and behavior. We show how text annotation of time series neuroimaging data using the system of Hierarchical Event Descriptors (HED; https://www.hedtags.org) can more adequately model the roles of both events and their ever-evolving context than current data annotation practice and can thereby facilitate data analysis, meta-analysis, and mega-analysis. Finally, we discuss ways in which the HED system must continue to expand to serve the evolving needs of neuroimaging research.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organization of memories remains unclear. Here, we designed a novel task-the Emotion Boundary Task-wherein participants viewed sequences of negative and neutral images surrounded by a color border. We manipulated perceptual context (border color), emotional valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create encoding events comprised of image sequences with a shared perceptual and/or emotional context. We measured memory for temporal order and subjectively remembered temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distance was remembered as longer for images spanning neutral-to-negative shifts-suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of this negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently presented negative (compared to neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral to negative events produce temporal dilation in memory, which may be relevant for adaptive emotional functioning.

Event Integration and Temporal Differentiation: How Hierarchical Knowledge Emerges in Hippocampal Subfields through Learning

Everyday life is composed of events organized by changes in contexts, with each event containing an unfolding sequence of occurrences. A major challenge facing our memory systems is how to integrate sequential occurrences within events while also maintaining their details and avoiding over-integration across different contexts. We asked if and how distinct hippocampal subfields come to hierarchically and, in parallel, represent both event context and subevent occurrences with learning. Female and male human participants viewed sequential events defined as sequences of objects superimposed on shared color frames while undergoing high-resolution fMRI. Importantly, these events were repeated to induce learning. Event segmentation, as indexed by increased reaction times at event boundaries, was observed in all repetitions. Temporal memory decisions were quicker for items from the same event compared to across different events, indicating that events shaped memory. With learning, hippocampal CA3 multivoxel activation patterns clustered to reflect the event context, with more clustering correlated with behavioral facilitation during event transitions. In contrast, in the dentate gyrus (DG), temporally proximal items that belonged to the same event became associated with more differentiated neural patterns. A computational model explained these results by dynamic inhibition in the DG. Additional similarity measures support the notion that CA3 clustered representations reflect shared voxel populations, while DG's distinct item representations reflect different voxel populations. These findings suggest an interplay between temporal differentiation in the DG and attractor dynamics in CA3. They advance our understanding of how knowledge is structured through integration and separation across time and context.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

Distinctive features of experiential time: Duration, speed and event density

William James's use of "time in passing" and "stream of thoughts" may be two sides of the same coin that emerge from the brain segmenting the continuous flow of information into discrete events. Herein, we investigated how the density of events affects two temporal experiences: the felt duration and speed of time. Using a temporal bisection task, participants classified seconds-long videos of naturalistic scenes as short or long (duration), or slow or fast (passage of time). Videos contained a varying number and type of events. We found that a large number of events lengthened subjective duration and accelerated the felt passage of time. Surprisingly, participants were also faster at estimating their felt passage of time compared to duration. The perception of duration scaled with duration and event density, whereas the felt passage of time scaled with the rate of change. Altogether, our results suggest that distinct mechanisms underlie these two experiential times.

The effect of hippocampal subfield damage on rapid temporal integration through statistical learning and associative inference

INTRODUCTION

The hippocampus (HPC) supports integration of information across time, often indexed by associative inference (AI) and statistical learning (SL) tasks. In AI, an indirect association between stimuli that never appeared together is inferred, whereas SL involves learning item relationships by extracting regularities across experiences. A recent model of hippocampal function (Schapiro et al., 2017) proposes that the HPC can support temporal integration in both paradigms through its two distinct pathways.

METHODS

We tested this models' predictions in four patients with varying degrees of bilateral HPC damage and matched healthy controls, with two patients with complementary damage to either the monosynaptic or trisynaptic pathway. During AI, participants studied overlapping paired associates (AB, BC) and their memory was tested for premise pairs (AB) and for inferred pairs (AC). During SL, participants passively viewed a continuous picture sequence that contained an underlying structure of triplets that later had to be recognized.

RESULTS

Binomial distributions were used to calculate above chance performance at the individual level. For AI, patients with focal HPC damage were impaired at inference but could correctly infer pairs above chance once premise pair acquisition was equated to controls; however, the patient with HPC and cortical damage showed severe impairment at recalling premise and inferred pairs, regardless of accounting for premise pair performance. For SL, none of the patients performed above chance, but notably neither did most controls.

CONCLUSIONS

Associative inference of indirect relationships can be intact with HPC damage to either hippocampal pathways or the HPC more broadly, provided premise pairs can first be formed. Inference may remain intact through residual HPC tissue supporting premise pair acquisition, and/or through extra-hippocampal structures supporting inference at retrieval. Clear conclusions about hippocampal contributions to SL are precluded by low performance in controls, which we caution is not dissimilar to previous amnesic studies using the same task. This complicates interpretations of studies claiming necessity of hippocampal contributions to SL and warrants the use of a common and reliable task before conclusions can be drawn.

Multiple timescales of temporal context in risky choice: Behavioral identification and relationships to physiological arousal

Context-dependence is fundamental to risky monetary decision-making. A growing body of evidence suggests that temporal context, or recent events, alters risk-taking at a minimum of three timescales: immediate (e.g. trial-by-trial), neighborhood (e.g. a group of consecutive trials), and global (e.g. task-level). To examine context effects, we created a novel monetary choice set with intentional temporal structure in which option values shifted between multiple levels of value magnitude ("contexts") several times over the course of the task. This structure allowed us to examine whether effects of each timescale were simultaneously present in risky choice behavior and the potential mechanistic role of arousal, an established correlate of risk-taking, in context-dependency. We found that risk-taking was sensitive to immediate, neighborhood, and global timescales: risk-taking decreased following large (vs. small) outcome amounts, increased following large positive (but not negative) shifts in context, and increased when cumulative earnings exceeded expectations. We quantified arousal with skin conductance responses, which were related to the global timescale, increasing with cumulative earnings, suggesting that physiological arousal captures a task-level assessment of performance. Our results both replicate and extend prior research by demonstrating that risky decision-making is consistently dynamic at multiple timescales and that the role of arousal in risk-taking extends to some, but not all timescales of context-dependence.

Rapid learning of temporal dependencies at multiple timescales

Our environment contains temporal information unfolding simultaneously at multiple timescales. How do we learn and represent these dynamic and overlapping information streams? We investigated these processes in a statistical learning paradigm with simultaneous short and long timescale contingencies. Human participants (N=96) played a game where they learned to quickly click on a target image when it appeared in one of 9 locations, in 8 different contexts. Across contexts, we manipulated the order of target locations: at a short timescale, the order of pairs of sequential locations in which the target appeared; at a longer timescale, the set of locations that appeared in the first vs. second half of the game. Participants periodically predicted the upcoming target location, and later performed similarity judgements comparing the games based on their order properties. Participants showed context dependent sensitivity to order information at both short and long timescales, with evidence of stronger learning for short timescales. We modeled the learning paradigm using a gated recurrent network trained to make immediate predictions, which demonstrated multilevel learning timecourses and patterns of sensitivity to the similarity structure of the games that mirrored human participants. The model grouped games with matching rule structure and dissociated games based on low-level order information more so than high-level order information. The work shows how humans and models can rapidly and concurrently acquire order information at different timescales.

More than a moment: What does it mean to call something an 'event'?

Experiences are stored in the mind as discrete mental units, or 'events,' which influence-and are influenced by-attention, learning, and memory. In this way, the notion of an 'event' is foundational to cognitive science. However, despite tremendous progress in understanding the behavioral and neural signatures of events, there is no agreed-upon definition of an event. Here, we discuss different theoretical frameworks of event perception and memory, noting what they can and cannot account for in the literature. We then highlight key aspects of events that we believe should be accounted for in theories of event processing--in particular, we argue that the structure and substance of events should be better reflected in our theories and paradigms. Finally, we discuss empirical gaps in the event cognition literature and what the future of event cognition research may look like.

Setting boundaries: Development of neural and behavioral event cognition in early childhood

The ongoing stream of sensory experience is so complex and ever-changing that we tend to parse this experience at "event boundaries," which structures and strengthens memory. Memory processes undergo profound change across early childhood. Whether young children also divide their ongoing processing along event boundaries, and if those boundaries relate to memory, could provide important insight into the development of memory systems. In Study 1, 4-7-year-old children and adults segmented a cartoon, and we tested their memory. Children's event boundaries were more variable than adults' and differed in location and consistency of agreement. Older children's event segmentation was more adult-like than younger children's, and children who segmented events more like adults had better memory for those events. In Study 2, we asked whether these developmental differences in event segmentation had their roots in distinct neural representations. A separate group of 4-8-year-old children watched the same cartoon while undergoing an fMRI scan. In the right hippocampus, greater pattern dissimilarity across event boundaries compared to within events was evident for both child and adult behavioral boundaries, suggesting children and adults share similar event cognition. However, the boundaries identified by a data-driven Hidden Markov Model found that a different brain region-the left and right angular gyrus-aligned only with event boundaries defined by children. Overall, these data suggest that children's event cognition is reasonably well-developed by age 4 but continues to become more adult-like across early childhood. RESEARCH HIGHLIGHTS: Adults naturally break their experience into events, which structures and strengthens memory, but less is known about children's event perception and memory. Study 1 had adults and children segment and remember events from an animated show, and Study 2 compared those segmentations to other children's fMRI data. Children show better recognition and temporal order memory and more adult-like event segmentation with age, and children who segment more like adults have better memory. Children's and adults' behavioral boundaries mapped onto pattern similarity differences in hippocampus, and children's behavioral boundaries matched a data-driven model's boundaries in angular gyrus.

Dynamic emotional states shape the episodic structure of memory

Human emotions fluctuate over time. However, it is unclear how these shifting emotional states influence the organization of episodic memory. Here, we examine how emotion dynamics transform experiences into memorable events. Using custom musical pieces and a dynamic emotion-tracking tool to elicit and measure temporal fluctuations in felt valence and arousal, our results demonstrate that memory is organized around emotional states. While listening to music, fluctuations between different emotional valences bias temporal encoding process toward memory integration or separation. Whereas a large absolute or negative shift in valence helps segment memories into episodes, a positive emotional shift binds sequential representations together. Both discrete and dynamic shifts in music-evoked valence and arousal also enhance delayed item and temporal source memory for concurrent neutral items, signaling the beginning of new emotional events. These findings are in line with the idea that the rise and fall of emotions can sculpt unfolding experiences into memories of meaningful events.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

Predictions transform memories: How expected versus unexpected events are integrated or separated in memory

Our brains constantly generate predictions about the environment based on prior knowledge. Many of the events we experience are consistent with these predictions, while others might be inconsistent with prior knowledge and thus violate our predictions. To guide future behavior, the memory system must be able to strengthen, transform, or add to existing knowledge based on the accuracy of our predictions. We synthesize recent evidence suggesting that when an event is consistent with our predictions, it leads to neural integration between related memories, which is associated with enhanced associative memory, as well as memory biases. Prediction errors, in turn, can promote both neural integration and separation, and lead to multiple mnemonic outcomes. We review these findings and how they interact with factors such as memory reactivation, prediction error strength, and task goals, to offer insight into what determines memory for events that violate our predictions. In doing so, this review brings together recent neural and behavioral research to advance our understanding of how predictions shape memory, and why.

Multiple routes to enhanced memory for emotionally relevant events

Events associated with aversive or rewarding outcomes are prioritized in memory. This memory boost is commonly attributed to the elicited affective response, closely linked to noradrenergic and dopaminergic modulation of hippocampal plasticity. Herein we review and compare this 'affect' mechanism to an additional, recently discovered, 'prediction' mechanism whereby memories are strengthened by the extent to which outcomes deviate from expectations, that is, by prediction errors (PEs). The mnemonic impact of PEs is separate from the affective outcome itself and has a distinct neural signature. While both routes enhance memory, these mechanisms are linked to different - and sometimes opposing - predictions for memory integration. We discuss new findings that highlight mechanisms by which emotional events strengthen, integrate, and segment memory.

Collective events and individual affect shape autobiographical memory

How do collective events shape how we remember our lives? We leveraged advances in natural language processing as well as a rich, longitudinal assessment of 1,000 Americans throughout 2020 to examine how memory is influenced by two prominent factors: surprise and emotion. Autobiographical memory for 2020 displayed a unique signature: There was a substantial bump in March, aligning with pandemic onset and lockdowns, consistent across three memory collections 1 y apart. We further investigated how emotion, using both immediate and retrieved measures, predicted the amount and content of autobiographical memory: Negative affect increased recall across all measures, whereas its more clinical indices, depression and posttraumatic stress disorder, selectively increased nonepisodic recall. Finally, in a separate cohort, we found pandemic news to be better remembered, surprising, and negative, while lockdowns compressed remembered time. Our work connects laboratory findings to the real world and delineates the effects of acute versus clinical signatures of negative emotion on memory.

Overharvesting in human patch foraging reflects rational structure learning and adaptive planning

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Inferring danger with minimal aversive experience

Learning about threats is crucial for survival and fundamentally rests upon Pavlovian conditioning. However, Pavlovian threat learning is largely limited to detecting known (or similar) threats and involves first-hand exposure to danger, which inevitably poses a risk of harm. We discuss how individuals leverage a rich repertoire of mnemonic processes that operate largely in safety and significantly expand our ability to recognize danger beyond Pavlovian threat associations. These processes result in complementary memories - acquired individually or through social interactions - that represent potential threats and the relational structure of our environment. The interplay between these memories allows danger to be inferred rather than directly learned, thereby flexibly protecting us from potential harm in novel situations despite minimal prior aversive experience.

Multi-view manifold learning of human brain-state trajectories

The complexity of the human brain gives the illusion that brain activity is intrinsically high-dimensional. Nonlinear dimensionality-reduction methods such as uniform manifold approximation and t-distributed stochastic neighbor embedding have been used for high-throughput biomedical data. However, they have not been used extensively for brain activity data such as those from functional magnetic resonance imaging (fMRI), primarily due to their inability to maintain dynamic structure. Here we introduce a nonlinear manifold learning method for time-series data-including those from fMRI-called temporal potential of heat-diffusion for affinity-based transition embedding (T-PHATE). In addition to recovering a low-dimensional intrinsic manifold geometry from time-series data, T-PHATE exploits the data's autocorrelative structure to faithfully denoise and unveil dynamic trajectories. We empirically validate T-PHATE on three fMRI datasets, showing that it greatly improves data visualization, classification, and segmentation of the data relative to several other state-of-the-art dimensionality-reduction benchmarks. These improvements suggest many potential applications of T-PHATE to other high-dimensional datasets of temporally diffuse processes.

What about "space" is important for episodic memory?

Early cognitive neuroscientific research revealed that the hippocampus is crucial for spatial navigation in rodents, and for autobiographical episodic memory in humans. Researchers quickly linked these streams to propose that the human hippocampus supports memory through its role in representing space, and research on the link between spatial cognition and episodic memory in humans has proliferated over the past several decades. Different researchers apply the term "spatial" in a variety of contexts, however, and it remains unclear what aspect of space may be critical to memory. Similarly, "episodic" has been defined and tested in different ways. Naturalistic assessment of spatial memory and episodic memory (i.e., episodic autobiographical memory) is required to unify the scale and biological relevance in comparisons of spatial and mnemonic processing. Limitations regarding the translation of rodent to human research, human ontogeny, and inter-individual variability require greater consideration in the interpretation of this literature. In this review, we outline the aspects of space that are (and are not) commonly linked to episodic memory, and then we discuss these dimensions through the lens of individual differences in naturalistic autobiographical memory. Future studies should carefully consider which aspect(s) of space are being linked to memory within the context of naturalistic human cognition. This article is categorized under: Psychology > Memory.

Mnemonic Content and Hippocampal Patterns Shape Judgments of Time

Our experience of time can feel dilated or compressed, rather than reflecting true "clock time." Although many contextual factors influence the subjective perception of time, it is unclear how memory accessibility plays a role in constructing our experience of and memory for time. Here, we used a combination of behavioral and functional MRI measures in healthy young adults (N = 147) to ask the question of how memory is incorporated into temporal duration judgments. Behaviorally, we found that event boundaries, which have been shown to disrupt ongoing memory integration processes, result in the temporal compression of duration judgments. Additionally, using a multivoxel pattern similarity analysis of functional MRI data, we found that greater temporal pattern change in the left hippocampus within individual trials was associated with longer duration judgments. Together, these data suggest that mnemonic processes play a role in constructing representations of time.

Neurodynamical Computing at the Information Boundaries of Intelligent Systems

Artificial intelligence has not achieved defining features of biological intelligence despite models boasting more parameters than neurons in the human brain. In this perspective article, we synthesize historical approaches to understanding intelligent systems and argue that methodological and epistemic biases in these fields can be resolved by shifting away from cognitivist brain-as-computer theories and recognizing that brains exist within large, interdependent living systems. Integrating the dynamical systems view of cognition with the massive distributed feedback of perceptual control theory highlights a theoretical gap in our understanding of nonreductive neural mechanisms. Cell assemblies-properly conceived as reentrant dynamical flows and not merely as identified groups of neurons-may fill that gap by providing a minimal supraneuronal level of organization that establishes a neurodynamical base layer for computation. By considering information streams from physical embodiment and situational embedding, we discuss this computational base layer in terms of conserved oscillatory and structural properties of cortical-hippocampal networks. Our synthesis of embodied cognition, based in dynamical systems and perceptual control, aims to bypass the neurosymbolic stalemates that have arisen in artificial intelligence, cognitive science, and computational neuroscience.

Neural event segmentation of continuous experience in human infants

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

A partially nested cortical hierarchy of neural states underlies event segmentation in the human brain

A fundamental aspect of human experience is that it is segmented into discrete events. This may be underpinned by transitions between distinct neural states. Using an innovative data-driven state segmentation method, we investigate how neural states are organized across the cortical hierarchy and where in the cortex neural state boundaries and perceived event boundaries overlap. Our results show that neural state boundaries are organized in a temporal cortical hierarchy, with short states in primary sensory regions, and long states in lateral and medial prefrontal cortex. State boundaries are shared within and between groups of brain regions that resemble well-known functional networks. Perceived event boundaries overlap with neural state boundaries across large parts of the cortical hierarchy, particularly when those state boundaries demarcate a strong transition or are shared between brain regions. Taken together, these findings suggest that a partially nested cortical hierarchy of neural states forms the basis of event segmentation.

From remembering to reconstruction: The transformative neural representation of episodic memory

Although memory has long been recognized as a generative process, neural research of memory in recent decades has been predominantly influenced by Tulving's "mental time traveling" perspective and focused on the reactivation and consolidation of encoded memory representations. With the development of multiple powerful analytical approaches to characterize the contents and formats of neural representations, recent studies are able to provide detailed examinations of the representations at various processing stages and have provided exciting new insights into the transformative nature of episodic memory. These studies have revealed the rapid, substantial, and continuous transformation of memory representation during the encoding, maintenance, consolidation, and retrieval of both single and multiple events, as well as event sequences. These transformations are characterized by the abstraction, integration, differentiation, and reorganization of memory representations, enabling the long-term retention and generalization of memory. These studies mark a significant shift in perspective from remembering to reconstruction, which might better reveal the nature of memory and its roles in supporting more effective learning, adaptive decision-making, and creative problem solving.

Predicting memory from the network structure of naturalistic events

When we remember events, we often do not only recall individual events, but also the connections between them. However, extant research has focused on how humans segment and remember discrete events from continuous input, with far less attention given to how the structure of connections between events impacts memory. Here we conduct a functional magnetic resonance imaging study in which participants watch and recall a series of realistic audiovisual narratives. By transforming narratives into networks of events, we demonstrate that more central events-those with stronger semantic or causal connections to other events-are better remembered. During encoding, central events evoke larger hippocampal event boundary responses associated with memory formation. During recall, high centrality is associated with stronger activation in cortical areas involved in episodic recollection, and more similar neural representations across individuals. Together, these results suggest that when humans encode and retrieve complex real-world experiences, the reliability and accessibility of memory representations is shaped by their location within a network of events.

A generalized cortical activity pattern at internally generated mental context boundaries during unguided narrative recall

Current theory and empirical studies suggest that humans segment continuous experiences into events based on the mismatch between predicted and actual sensory inputs; detection of these 'event boundaries' evokes transient neural responses. However, boundaries can also occur at transitions between internal mental states, without relevant external input changes. To what extent do such 'internal boundaries' share neural response properties with externally driven boundaries? We conducted an fMRI experiment where subjects watched a series of short movies and then verbally recalled the movies, unprompted, in the order of their choosing. During recall, transitions between movies thus constituted major boundaries between internal mental contexts, generated purely by subjects' unguided thoughts. Following the offset of each recalled movie, we observed stereotyped spatial activation patterns in the default mode network, especially the posterior medial cortex, consistent across different movie contents and even across the different tasks of movie watching and recall. Surprisingly, the between-movie boundary patterns did not resemble patterns at boundaries between events within a movie. Thus, major transitions between mental contexts elicit neural phenomena shared across internal and external modes and distinct from within-context event boundary detection, potentially reflecting a cognitive state related to the flushing and reconfiguration of situation models.

Effects of spatial boundaries on episodic memory development

Children's spatial mapping starts out particularly sensitive to 3D wall-like boundaries and develops over early childhood to flexibly include other boundary types. This study investigated whether spatial boundaries influence children's episodic memory, as in adults, and whether this effect is modulated by boundary type. Eighty-one Korean children (34 girls, 36-84 months old) re-enacted a sequence of three discrete hiding events within a space containing one of three boundaries: 3D wall, aligned objects, or 2D line. Children's memory of events occurring on one side of the boundary developed earlier than those that crossed the boundary. At first, this interaction only applied to the 3D wall and extended to other boundary types with age, suggesting that children's changing spatial representations influence their episodic memory development.

Switching task sets creates event boundaries in memory

People segregate continuously unfolding experiences into discrete events in memory. This process, known as event segmentation, results in better memory for the temporal order of experiences within an event and expands subjective temporal distance for items encoded across event boundaries. Previous research has suggested that the creation of event boundaries is driven by (typically unpredicted) changes in external stimulation, though many prior studies have confounded a change in bottom-up input with a concurrent change in task goal. This raises the question of whether event segmentation can be triggered by the endogenous cognitive control processes involved in switching task sets, independent of changes in bottom-up stimulation. We investigated this question by embedding task set changes during encoding of a series of trial-unique images, and comparing subsequent temporal order and distance memory for item pairs encoded across a change in task set with item pairs encoded within the same task set. Across five experiments, we demonstrate that both cued and voluntary task set changes are sufficient to create event boundaries, while ruling out potential confounding effects of shifts in stimulus set, response set, task cues, and task difficulty. Thus, internal control processes are a key determinant of segmenting episodic memories, and task set updating can trigger event segmentation independent of any externally induced, perceptual or task-based prediction error.

Mixing memory and desire: How memory reactivation supports deliberative decision-making

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affects deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted based on their uncertainty. We close by reviewing related findings and identifying areas for further research. This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition Neuroscience > Computation.

Touching events predict human action segmentation in brain and behavior

Recognizing the actions of others depends on segmentation into meaningful events. After decades of research in this area, it remains still unclear how humans do this and which brain areas support underlying processes. Here we show that a computer vision-based model of touching and untouching events can predict human behavior in segmenting object manipulation actions with high accuracy. Using this computational model and functional Magnetic Resonance Imaging (fMRI), we pinpoint the neural networks underlying this segmentation behavior during an implicit action observation task. Segmentation was announced by a strong increase of visual activity at touching events followed by the engagement of frontal, hippocampal and insula regions, signaling updating expectation at subsequent untouching events. Brain activity and behavior show that touching-untouching motifs are critical features for identifying the key elements of actions including object manipulations.

Events structure information accessibility less in children than adults

To manage the onslaught of continuously unfolding information in our complex environments, we adults are known to carve up our continuous experience into meaningful events, a process referred to as event segmentation. This segmentation directly shapes how our everyday experiences are construed: content experienced within an event is held mentally in an accessible state, which is then dropped after an event boundary. The greater accessibility of event-specific information has been shown to influence-at its most basic level-how information is processed and remembered. However, it is as yet unknown if accessibility is similarly influenced by event boundaries in children, who are still developing the working memory capacity and semantic knowledge thought to support event segmentation. Here, we tested seven- to nine-year-old children's and adults' recognition of objects experienced either within or across event boundaries of two cartoons. We found that children and adults were both more accurate and faster to correctly recognize objects that last occurred within events versus across event boundaries. We, however, additionally observed an interaction such that children's access to recent experience was less influenced by event boundaries than adults'. Thus, while the spontaneous segmentation of complex events emerges by middle childhood, event structure shapes the active contents of children's minds less reliably than adults'.

Replay in minds and machines

Experience-related brain activity patterns reactivate during sleep, wakeful rest, and brief pauses from active behavior. In parallel, machine learning research has found that experience replay can lead to substantial performance improvements in artificial agents. Together, these lines of research suggest replay has a variety of computational benefits for decision-making and learning. Here, we provide an overview of putative computational functions of replay as suggested by machine learning and neuroscientific research. We show that replay can lead to faster learning, less forgetting, reorganization or augmentation of experiences, and support planning and generalization. In addition, we highlight the benefits of reactivating abstracted internal representations rather than veridical memories, and discuss how replay could provide a mechanism to build internal representations that improve learning and decision-making.

Human Representation Learning

The central theme of this review is the dynamic interaction between information selection and learning. We pose a fundamental question about this interaction: How do we learn what features of our experiences are worth learning about? In humans, this process depends on attention and memory, two cognitive functions that together constrain representations of the world to features that are relevant for goal attainment. Recent evidence suggests that the representations shaped by attention and memory are themselves inferred from experience with each task. We review this evidence and place it in the context of work that has explicitly characterized representation learning as statistical inference. We discuss how inference can be scaled to real-world decisions by approximating beliefs based on a small number of experiences. Finally, we highlight some implications of this inference process for human decision-making in social environments.