Reconciling shared versus context-specific information in a neural network model of latent causes

It has been proposed that, when processing a stream of events, humans divide their experiences in terms of inferred latent causes (LCs) to support context-dependent learning. However, when shared structure is present across contexts, it is still unclear how the "splitting" of LCs and learning of shared structure can be simultaneously achieved. Here, we present the Latent Cause Network (LCNet), a neural network model of LC inference. Through learning, it naturally stores structure that is shared across tasks in the network weights. Additionally, it represents context-specific structure using a context module, controlled by a Bayesian nonparametric inference algorithm, which assigns a unique context vector for each inferred LC. Across three simulations, we found that LCNet could (1) extract shared structure across LCs in a function learning task while avoiding catastrophic interference, (2) capture human data on curriculum effects in schema learning, and (3) infer the underlying event structure when processing naturalistic videos of daily events. Overall, these results demonstrate a computationally feasible approach to reconciling shared structure and context-specific structure in a model of LCs that is scalable from laboratory experiment settings to naturalistic settings.

Do actions structure auditory memory? Action-based event segmentation effects on sensory responses, pupil dilation and sequential memory

Our actions shape our everyday experience: what we experience, how we perceive, and remember it are deeply affected by how we interact with the world. Performing an action to deliver a stimulus engages neurophysiological processes which are reflected in the modulation of sensory and pupil responses. We hypothesized that these processes shape memory encoding, parsing the experience by grouping self- and externally generated stimuli into differentiated events. Participants encoded sound sequences, in which either the first or last few sounds were self-generated and the rest externally generated. We tested recall of the sequential order of sounds that had originated from the same (within event) or different sources (across events). Memory performance was not higher for within-event sounds, suggesting that actions did not structure the memory representation. However, during encoding, we observed the expected electrophysiological response attenuation for self-generated sounds, together with increased pupil dilation triggered by actions. Moreover, at the boundary between events, physiological responses to the first sound from the new source were influenced by the direction of the source switch. Our results suggest that introducing actions creates a stronger contextual shift than removing them, even though actions do not directly contribute to memory performance. This study contributes to our understanding of how interacting with sensory input shapes experiences by exploring the relationships between action effects on sensory responses, pupil dilation, and memory encoding. Importantly, it challenges the notion of a meaningful contribution from low-level neurophysiological mechanisms associated with action execution in the modulation of the self-generation effect.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organisation of memories remains unclear. Here, we designed a novel task - the Emotion Boundary Task - wherein participants viewed sequences of negative and neutral images surrounded by a colour border. We manipulated perceptual context (border colour), emotional-picture valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create events with a shared perceptual and/or emotional context. We measured memory for temporal order and temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distances were remembered as longer for images spanning neutral-to-negative shifts - suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently-presented negative (versus neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral-to-negative events produce temporal dilation in memory, with implications for adaptive emotional functioning.

Events in context-The HED framework for the study of brain, experience and behavior

The brain is a complex dynamic system whose current state is inextricably coupled to awareness of past, current, and anticipated future threats and opportunities that continually affect awareness and behavioral goals and decisions. Brain activity is driven on multiple time scales by an ever-evolving flow of sensory, proprioceptive, and idiothetic experience. Neuroimaging experiments seek to isolate and focus on some aspect of these complex dynamics to better understand how human experience, cognition, behavior, and health are supported by brain activity. Here we consider an event-related data modeling approach that seeks to parse experience and behavior into a set of time-delimited events. We distinguish between event processes themselves, that unfold through time, and event markers that record the experiment timeline latencies of event onset, offset, and any other event phase transitions. Precise descriptions of experiment events (sensory, motor, or other) allow participant experience and behavior to be interpreted in the context either of the event itself or of all or any experiment events. We discuss how events in neuroimaging experiments have been, are currently, and should best be identified and represented with emphasis on the importance of modeling both events and event context for meaningful interpretation of relationships between brain dynamics, experience, and behavior. We show how text annotation of time series neuroimaging data using the system of Hierarchical Event Descriptors (HED; https://www.hedtags.org) can more adequately model the roles of both events and their ever-evolving context than current data annotation practice and can thereby facilitate data analysis, meta-analysis, and mega-analysis. Finally, we discuss ways in which the HED system must continue to expand to serve the evolving needs of neuroimaging research.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organization of memories remains unclear. Here, we designed a novel task-the Emotion Boundary Task-wherein participants viewed sequences of negative and neutral images surrounded by a color border. We manipulated perceptual context (border color), emotional valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create encoding events comprised of image sequences with a shared perceptual and/or emotional context. We measured memory for temporal order and subjectively remembered temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distance was remembered as longer for images spanning neutral-to-negative shifts-suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of this negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently presented negative (compared to neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral to negative events produce temporal dilation in memory, which may be relevant for adaptive emotional functioning.

Event Integration and Temporal Differentiation: How Hierarchical Knowledge Emerges in Hippocampal Subfields through Learning

Everyday life is composed of events organized by changes in contexts, with each event containing an unfolding sequence of occurrences. A major challenge facing our memory systems is how to integrate sequential occurrences within events while also maintaining their details and avoiding over-integration across different contexts. We asked if and how distinct hippocampal subfields come to hierarchically and, in parallel, represent both event context and subevent occurrences with learning. Female and male human participants viewed sequential events defined as sequences of objects superimposed on shared color frames while undergoing high-resolution fMRI. Importantly, these events were repeated to induce learning. Event segmentation, as indexed by increased reaction times at event boundaries, was observed in all repetitions. Temporal memory decisions were quicker for items from the same event compared to across different events, indicating that events shaped memory. With learning, hippocampal CA3 multivoxel activation patterns clustered to reflect the event context, with more clustering correlated with behavioral facilitation during event transitions. In contrast, in the dentate gyrus (DG), temporally proximal items that belonged to the same event became associated with more differentiated neural patterns. A computational model explained these results by dynamic inhibition in the DG. Additional similarity measures support the notion that CA3 clustered representations reflect shared voxel populations, while DG's distinct item representations reflect different voxel populations. These findings suggest an interplay between temporal differentiation in the DG and attractor dynamics in CA3. They advance our understanding of how knowledge is structured through integration and separation across time and context.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

Distinctive features of experiential time: Duration, speed and event density

William James's use of "time in passing" and "stream of thoughts" may be two sides of the same coin that emerge from the brain segmenting the continuous flow of information into discrete events. Herein, we investigated how the density of events affects two temporal experiences: the felt duration and speed of time. Using a temporal bisection task, participants classified seconds-long videos of naturalistic scenes as short or long (duration), or slow or fast (passage of time). Videos contained a varying number and type of events. We found that a large number of events lengthened subjective duration and accelerated the felt passage of time. Surprisingly, participants were also faster at estimating their felt passage of time compared to duration. The perception of duration scaled with duration and event density, whereas the felt passage of time scaled with the rate of change. Altogether, our results suggest that distinct mechanisms underlie these two experiential times.

The effect of hippocampal subfield damage on rapid temporal integration through statistical learning and associative inference

INTRODUCTION

The hippocampus (HPC) supports integration of information across time, often indexed by associative inference (AI) and statistical learning (SL) tasks. In AI, an indirect association between stimuli that never appeared together is inferred, whereas SL involves learning item relationships by extracting regularities across experiences. A recent model of hippocampal function (Schapiro et al., 2017) proposes that the HPC can support temporal integration in both paradigms through its two distinct pathways.

METHODS

We tested this models' predictions in four patients with varying degrees of bilateral HPC damage and matched healthy controls, with two patients with complementary damage to either the monosynaptic or trisynaptic pathway. During AI, participants studied overlapping paired associates (AB, BC) and their memory was tested for premise pairs (AB) and for inferred pairs (AC). During SL, participants passively viewed a continuous picture sequence that contained an underlying structure of triplets that later had to be recognized.

RESULTS

Binomial distributions were used to calculate above chance performance at the individual level. For AI, patients with focal HPC damage were impaired at inference but could correctly infer pairs above chance once premise pair acquisition was equated to controls; however, the patient with HPC and cortical damage showed severe impairment at recalling premise and inferred pairs, regardless of accounting for premise pair performance. For SL, none of the patients performed above chance, but notably neither did most controls.

CONCLUSIONS

Associative inference of indirect relationships can be intact with HPC damage to either hippocampal pathways or the HPC more broadly, provided premise pairs can first be formed. Inference may remain intact through residual HPC tissue supporting premise pair acquisition, and/or through extra-hippocampal structures supporting inference at retrieval. Clear conclusions about hippocampal contributions to SL are precluded by low performance in controls, which we caution is not dissimilar to previous amnesic studies using the same task. This complicates interpretations of studies claiming necessity of hippocampal contributions to SL and warrants the use of a common and reliable task before conclusions can be drawn.

Multiple timescales of temporal context in risky choice: Behavioral identification and relationships to physiological arousal

Context-dependence is fundamental to risky monetary decision-making. A growing body of evidence suggests that temporal context, or recent events, alters risk-taking at a minimum of three timescales: immediate (e.g. trial-by-trial), neighborhood (e.g. a group of consecutive trials), and global (e.g. task-level). To examine context effects, we created a novel monetary choice set with intentional temporal structure in which option values shifted between multiple levels of value magnitude ("contexts") several times over the course of the task. This structure allowed us to examine whether effects of each timescale were simultaneously present in risky choice behavior and the potential mechanistic role of arousal, an established correlate of risk-taking, in context-dependency. We found that risk-taking was sensitive to immediate, neighborhood, and global timescales: risk-taking decreased following large (vs. small) outcome amounts, increased following large positive (but not negative) shifts in context, and increased when cumulative earnings exceeded expectations. We quantified arousal with skin conductance responses, which were related to the global timescale, increasing with cumulative earnings, suggesting that physiological arousal captures a task-level assessment of performance. Our results both replicate and extend prior research by demonstrating that risky decision-making is consistently dynamic at multiple timescales and that the role of arousal in risk-taking extends to some, but not all timescales of context-dependence.

Rapid learning of temporal dependencies at multiple timescales

Our environment contains temporal information unfolding simultaneously at multiple timescales. How do we learn and represent these dynamic and overlapping information streams? We investigated these processes in a statistical learning paradigm with simultaneous short and long timescale contingencies. Human participants (N=96) played a game where they learned to quickly click on a target image when it appeared in one of 9 locations, in 8 different contexts. Across contexts, we manipulated the order of target locations: at a short timescale, the order of pairs of sequential locations in which the target appeared; at a longer timescale, the set of locations that appeared in the first vs. second half of the game. Participants periodically predicted the upcoming target location, and later performed similarity judgements comparing the games based on their order properties. Participants showed context dependent sensitivity to order information at both short and long timescales, with evidence of stronger learning for short timescales. We modeled the learning paradigm using a gated recurrent network trained to make immediate predictions, which demonstrated multilevel learning timecourses and patterns of sensitivity to the similarity structure of the games that mirrored human participants. The model grouped games with matching rule structure and dissociated games based on low-level order information more so than high-level order information. The work shows how humans and models can rapidly and concurrently acquire order information at different timescales.

More than a moment: What does it mean to call something an 'event'?

Experiences are stored in the mind as discrete mental units, or 'events,' which influence-and are influenced by-attention, learning, and memory. In this way, the notion of an 'event' is foundational to cognitive science. However, despite tremendous progress in understanding the behavioral and neural signatures of events, there is no agreed-upon definition of an event. Here, we discuss different theoretical frameworks of event perception and memory, noting what they can and cannot account for in the literature. We then highlight key aspects of events that we believe should be accounted for in theories of event processing--in particular, we argue that the structure and substance of events should be better reflected in our theories and paradigms. Finally, we discuss empirical gaps in the event cognition literature and what the future of event cognition research may look like.

Setting boundaries: Development of neural and behavioral event cognition in early childhood

The ongoing stream of sensory experience is so complex and ever-changing that we tend to parse this experience at "event boundaries," which structures and strengthens memory. Memory processes undergo profound change across early childhood. Whether young children also divide their ongoing processing along event boundaries, and if those boundaries relate to memory, could provide important insight into the development of memory systems. In Study 1, 4-7-year-old children and adults segmented a cartoon, and we tested their memory. Children's event boundaries were more variable than adults' and differed in location and consistency of agreement. Older children's event segmentation was more adult-like than younger children's, and children who segmented events more like adults had better memory for those events. In Study 2, we asked whether these developmental differences in event segmentation had their roots in distinct neural representations. A separate group of 4-8-year-old children watched the same cartoon while undergoing an fMRI scan. In the right hippocampus, greater pattern dissimilarity across event boundaries compared to within events was evident for both child and adult behavioral boundaries, suggesting children and adults share similar event cognition. However, the boundaries identified by a data-driven Hidden Markov Model found that a different brain region-the left and right angular gyrus-aligned only with event boundaries defined by children. Overall, these data suggest that children's event cognition is reasonably well-developed by age 4 but continues to become more adult-like across early childhood. RESEARCH HIGHLIGHTS: Adults naturally break their experience into events, which structures and strengthens memory, but less is known about children's event perception and memory. Study 1 had adults and children segment and remember events from an animated show, and Study 2 compared those segmentations to other children's fMRI data. Children show better recognition and temporal order memory and more adult-like event segmentation with age, and children who segment more like adults have better memory. Children's and adults' behavioral boundaries mapped onto pattern similarity differences in hippocampus, and children's behavioral boundaries matched a data-driven model's boundaries in angular gyrus.

Dynamic emotional states shape the episodic structure of memory

Human emotions fluctuate over time. However, it is unclear how these shifting emotional states influence the organization of episodic memory. Here, we examine how emotion dynamics transform experiences into memorable events. Using custom musical pieces and a dynamic emotion-tracking tool to elicit and measure temporal fluctuations in felt valence and arousal, our results demonstrate that memory is organized around emotional states. While listening to music, fluctuations between different emotional valences bias temporal encoding process toward memory integration or separation. Whereas a large absolute or negative shift in valence helps segment memories into episodes, a positive emotional shift binds sequential representations together. Both discrete and dynamic shifts in music-evoked valence and arousal also enhance delayed item and temporal source memory for concurrent neutral items, signaling the beginning of new emotional events. These findings are in line with the idea that the rise and fall of emotions can sculpt unfolding experiences into memories of meaningful events.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

Predictions transform memories: How expected versus unexpected events are integrated or separated in memory

Our brains constantly generate predictions about the environment based on prior knowledge. Many of the events we experience are consistent with these predictions, while others might be inconsistent with prior knowledge and thus violate our predictions. To guide future behavior, the memory system must be able to strengthen, transform, or add to existing knowledge based on the accuracy of our predictions. We synthesize recent evidence suggesting that when an event is consistent with our predictions, it leads to neural integration between related memories, which is associated with enhanced associative memory, as well as memory biases. Prediction errors, in turn, can promote both neural integration and separation, and lead to multiple mnemonic outcomes. We review these findings and how they interact with factors such as memory reactivation, prediction error strength, and task goals, to offer insight into what determines memory for events that violate our predictions. In doing so, this review brings together recent neural and behavioral research to advance our understanding of how predictions shape memory, and why.

Multiple routes to enhanced memory for emotionally relevant events

Events associated with aversive or rewarding outcomes are prioritized in memory. This memory boost is commonly attributed to the elicited affective response, closely linked to noradrenergic and dopaminergic modulation of hippocampal plasticity. Herein we review and compare this 'affect' mechanism to an additional, recently discovered, 'prediction' mechanism whereby memories are strengthened by the extent to which outcomes deviate from expectations, that is, by prediction errors (PEs). The mnemonic impact of PEs is separate from the affective outcome itself and has a distinct neural signature. While both routes enhance memory, these mechanisms are linked to different - and sometimes opposing - predictions for memory integration. We discuss new findings that highlight mechanisms by which emotional events strengthen, integrate, and segment memory.

Collective events and individual affect shape autobiographical memory

How do collective events shape how we remember our lives? We leveraged advances in natural language processing as well as a rich, longitudinal assessment of 1,000 Americans throughout 2020 to examine how memory is influenced by two prominent factors: surprise and emotion. Autobiographical memory for 2020 displayed a unique signature: There was a substantial bump in March, aligning with pandemic onset and lockdowns, consistent across three memory collections 1 y apart. We further investigated how emotion, using both immediate and retrieved measures, predicted the amount and content of autobiographical memory: Negative affect increased recall across all measures, whereas its more clinical indices, depression and posttraumatic stress disorder, selectively increased nonepisodic recall. Finally, in a separate cohort, we found pandemic news to be better remembered, surprising, and negative, while lockdowns compressed remembered time. Our work connects laboratory findings to the real world and delineates the effects of acute versus clinical signatures of negative emotion on memory.

Overharvesting in human patch foraging reflects rational structure learning and adaptive planning

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Inferring danger with minimal aversive experience

Learning about threats is crucial for survival and fundamentally rests upon Pavlovian conditioning. However, Pavlovian threat learning is largely limited to detecting known (or similar) threats and involves first-hand exposure to danger, which inevitably poses a risk of harm. We discuss how individuals leverage a rich repertoire of mnemonic processes that operate largely in safety and significantly expand our ability to recognize danger beyond Pavlovian threat associations. These processes result in complementary memories - acquired individually or through social interactions - that represent potential threats and the relational structure of our environment. The interplay between these memories allows danger to be inferred rather than directly learned, thereby flexibly protecting us from potential harm in novel situations despite minimal prior aversive experience.

Multi-view manifold learning of human brain-state trajectories

The complexity of the human brain gives the illusion that brain activity is intrinsically high-dimensional. Nonlinear dimensionality-reduction methods such as uniform manifold approximation and t-distributed stochastic neighbor embedding have been used for high-throughput biomedical data. However, they have not been used extensively for brain activity data such as those from functional magnetic resonance imaging (fMRI), primarily due to their inability to maintain dynamic structure. Here we introduce a nonlinear manifold learning method for time-series data-including those from fMRI-called temporal potential of heat-diffusion for affinity-based transition embedding (T-PHATE). In addition to recovering a low-dimensional intrinsic manifold geometry from time-series data, T-PHATE exploits the data's autocorrelative structure to faithfully denoise and unveil dynamic trajectories. We empirically validate T-PHATE on three fMRI datasets, showing that it greatly improves data visualization, classification, and segmentation of the data relative to several other state-of-the-art dimensionality-reduction benchmarks. These improvements suggest many potential applications of T-PHATE to other high-dimensional datasets of temporally diffuse processes.

What about "space" is important for episodic memory?

Early cognitive neuroscientific research revealed that the hippocampus is crucial for spatial navigation in rodents, and for autobiographical episodic memory in humans. Researchers quickly linked these streams to propose that the human hippocampus supports memory through its role in representing space, and research on the link between spatial cognition and episodic memory in humans has proliferated over the past several decades. Different researchers apply the term "spatial" in a variety of contexts, however, and it remains unclear what aspect of space may be critical to memory. Similarly, "episodic" has been defined and tested in different ways. Naturalistic assessment of spatial memory and episodic memory (i.e., episodic autobiographical memory) is required to unify the scale and biological relevance in comparisons of spatial and mnemonic processing. Limitations regarding the translation of rodent to human research, human ontogeny, and inter-individual variability require greater consideration in the interpretation of this literature. In this review, we outline the aspects of space that are (and are not) commonly linked to episodic memory, and then we discuss these dimensions through the lens of individual differences in naturalistic autobiographical memory. Future studies should carefully consider which aspect(s) of space are being linked to memory within the context of naturalistic human cognition. This article is categorized under: Psychology > Memory.

Mnemonic Content and Hippocampal Patterns Shape Judgments of Time

Our experience of time can feel dilated or compressed, rather than reflecting true "clock time." Although many contextual factors influence the subjective perception of time, it is unclear how memory accessibility plays a role in constructing our experience of and memory for time. Here, we used a combination of behavioral and functional MRI measures in healthy young adults (N = 147) to ask the question of how memory is incorporated into temporal duration judgments. Behaviorally, we found that event boundaries, which have been shown to disrupt ongoing memory integration processes, result in the temporal compression of duration judgments. Additionally, using a multivoxel pattern similarity analysis of functional MRI data, we found that greater temporal pattern change in the left hippocampus within individual trials was associated with longer duration judgments. Together, these data suggest that mnemonic processes play a role in constructing representations of time.

Neurodynamical Computing at the Information Boundaries of Intelligent Systems

Artificial intelligence has not achieved defining features of biological intelligence despite models boasting more parameters than neurons in the human brain. In this perspective article, we synthesize historical approaches to understanding intelligent systems and argue that methodological and epistemic biases in these fields can be resolved by shifting away from cognitivist brain-as-computer theories and recognizing that brains exist within large, interdependent living systems. Integrating the dynamical systems view of cognition with the massive distributed feedback of perceptual control theory highlights a theoretical gap in our understanding of nonreductive neural mechanisms. Cell assemblies—properly conceived as reentrant dynamical flows and not merely as identified groups of neurons—may fill that gap by providing a minimal supraneuronal level of organization that establishes a neurodynamical base layer for computation. By considering information streams from physical embodiment and situational embedding, we discuss this computational base layer in terms of conserved oscillatory and structural properties of cortical-hippocampal networks. Our synthesis of embodied cognition, based in dynamical systems and perceptual control, aims to bypass the neurosymbolic stalemates that have arisen in artificial intelligence, cognitive science, and computational neuroscience.

Neural event segmentation of continuous experience in human infants

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

From remembering to reconstruction: The transformative neural representation of episodic memory

Although memory has long been recognized as a generative process, neural research of memory in recent decades has been predominantly influenced by Tulving's "mental time traveling" perspective and focused on the reactivation and consolidation of encoded memory representations. With the development of multiple powerful analytical approaches to characterize the contents and formats of neural representations, recent studies are able to provide detailed examinations of the representations at various processing stages and have provided exciting new insights into the transformative nature of episodic memory. These studies have revealed the rapid, substantial, and continuous transformation of memory representation during the encoding, maintenance, consolidation, and retrieval of both single and multiple events, as well as event sequences. These transformations are characterized by the abstraction, integration, differentiation, and reorganization of memory representations, enabling the long-term retention and generalization of memory. These studies mark a significant shift in perspective from remembering to reconstruction, which might better reveal the nature of memory and its roles in supporting more effective learning, adaptive decision-making, and creative problem solving.

Predicting memory from the network structure of naturalistic events

When we remember events, we often do not only recall individual events, but also the connections between them. However, extant research has focused on how humans segment and remember discrete events from continuous input, with far less attention given to how the structure of connections between events impacts memory. Here we conduct a functional magnetic resonance imaging study in which participants watch and recall a series of realistic audiovisual narratives. By transforming narratives into networks of events, we demonstrate that more central events-those with stronger semantic or causal connections to other events-are better remembered. During encoding, central events evoke larger hippocampal event boundary responses associated with memory formation. During recall, high centrality is associated with stronger activation in cortical areas involved in episodic recollection, and more similar neural representations across individuals. Together, these results suggest that when humans encode and retrieve complex real-world experiences, the reliability and accessibility of memory representations is shaped by their location within a network of events.

A generalized cortical activity pattern at internally generated mental context boundaries during unguided narrative recall

Current theory and empirical studies suggest that humans segment continuous experiences into events based on the mismatch between predicted and actual sensory inputs; detection of these ‘event boundaries’ evokes transient neural responses. However, boundaries can also occur at transitions between internal mental states, without relevant external input changes. To what extent do such ‘internal boundaries’ share neural response properties with externally driven boundaries? We conducted an fMRI experiment where subjects watched a series of short movies and then verbally recalled the movies, unprompted, in the order of their choosing. During recall, transitions between movies thus constituted major boundaries between internal mental contexts, generated purely by subjects’ unguided thoughts. Following the offset of each recalled movie, we observed stereotyped spatial activation patterns in the default mode network, especially the posterior medial cortex, consistent across different movie contents and even across the different tasks of movie watching and recall. Surprisingly, the between-movie boundary patterns did not resemble patterns at boundaries between events within a movie. Thus, major transitions between mental contexts elicit neural phenomena shared across internal and external modes and distinct from within-context event boundary detection, potentially reflecting a cognitive state related to the flushing and reconfiguration of situation models.

Effects of spatial boundaries on episodic memory development

Children's spatial mapping starts out particularly sensitive to 3D wall-like boundaries and develops over early childhood to flexibly include other boundary types. This study investigated whether spatial boundaries influence children's episodic memory, as in adults, and whether this effect is modulated by boundary type. Eighty-one Korean children (34 girls, 36-84 months old) re-enacted a sequence of three discrete hiding events within a space containing one of three boundaries: 3D wall, aligned objects, or 2D line. Children's memory of events occurring on one side of the boundary developed earlier than those that crossed the boundary. At first, this interaction only applied to the 3D wall and extended to other boundary types with age, suggesting that children's changing spatial representations influence their episodic memory development.

Switching task sets creates event boundaries in memory

People segregate continuously unfolding experiences into discrete events in memory. This process, known as event segmentation, results in better memory for the temporal order of experiences within an event and expands subjective temporal distance for items encoded across event boundaries. Previous research has suggested that the creation of event boundaries is driven by (typically unpredicted) changes in external stimulation, though many prior studies have confounded a change in bottom-up input with a concurrent change in task goal. This raises the question of whether event segmentation can be triggered by the endogenous cognitive control processes involved in switching task sets, independent of changes in bottom-up stimulation. We investigated this question by embedding task set changes during encoding of a series of trial-unique images, and comparing subsequent temporal order and distance memory for item pairs encoded across a change in task set with item pairs encoded within the same task set. Across five experiments, we demonstrate that both cued and voluntary task set changes are sufficient to create event boundaries, while ruling out potential confounding effects of shifts in stimulus set, response set, task cues, and task difficulty. Thus, internal control processes are a key determinant of segmenting episodic memories, and task set updating can trigger event segmentation independent of any externally induced, perceptual or task-based prediction error.

Mixing memory and desire: How memory reactivation supports deliberative decision-making

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affects deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted based on their uncertainty. We close by reviewing related findings and identifying areas for further research. This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition Neuroscience > Computation.

Touching events predict human action segmentation in brain and behavior

Recognizing the actions of others depends on segmentation into meaningful events. After decades of research in this area, it remains still unclear how humans do this and which brain areas support underlying processes. Here we show that a computer vision-based model of touching and untouching events can predict human behavior in segmenting object manipulation actions with high accuracy. Using this computational model and functional Magnetic Resonance Imaging (fMRI), we pinpoint the neural networks underlying this segmentation behavior during an implicit action observation task. Segmentation was announced by a strong increase of visual activity at touching events followed by the engagement of frontal, hippocampal and insula regions, signaling updating expectation at subsequent untouching events. Brain activity and behavior show that touching-untouching motifs are critical features for identifying the key elements of actions including object manipulations.

Events structure information accessibility less in children than adults

To manage the onslaught of continuously unfolding information in our complex environments, we adults are known to carve up our continuous experience into meaningful events, a process referred to as event segmentation. This segmentation directly shapes how our everyday experiences are construed: content experienced within an event is held mentally in an accessible state, which is then dropped after an event boundary. The greater accessibility of event-specific information has been shown to influence-at its most basic level-how information is processed and remembered. However, it is as yet unknown if accessibility is similarly influenced by event boundaries in children, who are still developing the working memory capacity and semantic knowledge thought to support event segmentation. Here, we tested seven- to nine-year-old children's and adults' recognition of objects experienced either within or across event boundaries of two cartoons. We found that children and adults were both more accurate and faster to correctly recognize objects that last occurred within events versus across event boundaries. We, however, additionally observed an interaction such that children's access to recent experience was less influenced by event boundaries than adults'. Thus, while the spontaneous segmentation of complex events emerges by middle childhood, event structure shapes the active contents of children's minds less reliably than adults'.

Replay in minds and machines

Experience-related brain activity patterns reactivate during sleep, wakeful rest, and brief pauses from active behavior. In parallel, machine learning research has found that experience replay can lead to substantial performance improvements in artificial agents. Together, these lines of research suggest replay has a variety of computational benefits for decision-making and learning. Here, we provide an overview of putative computational functions of replay as suggested by machine learning and neuroscientific research. We show that replay can lead to faster learning, less forgetting, reorganization or augmentation of experiences, and support planning and generalization. In addition, we highlight the benefits of reactivating abstracted internal representations rather than veridical memories, and discuss how replay could provide a mechanism to build internal representations that improve learning and decision-making.

Human Representation Learning

The central theme of this review is the dynamic interaction between information selection and learning. We pose a fundamental question about this interaction: How do we learn what features of our experiences are worth learning about? In humans, this process depends on attention and memory, two cognitive functions that together constrain representations of the world to features that are relevant for goal attainment. Recent evidence suggests that the representations shaped by attention and memory are themselves inferred from experience with each task. We review this evidence and place it in the context of work that has explicitly characterized representation learning as statistical inference. We discuss how inference can be scaled to real-world decisions by approximating beliefs based on a small number of experiences. Finally, we highlight some implications of this inference process for human decision-making in social environments.

Towards Strong AI

Strong AI—artificial intelligence that is in all respects at least as intelligent as humans—is still out of reach. Current AI lacks common sense, that is, it is not able to infer, understand, or explain the hidden processes, forces, and causes behind data. Main stream machine learning research on deep artificial neural networks (ANNs) may even be characterized as being behavioristic. In contrast, various sources of evidence from cognitive science suggest that human brains engage in the active development of compositional generative predictive models (CGPMs) from their self-generated sensorimotor experiences. Guided by evolutionarily-shaped inductive learning and information processing biases, they exhibit the tendency to organize the gathered experiences into event-predictive encodings. Meanwhile, they infer and optimize behavior and attention by means of both epistemic- and homeostasis-oriented drives. I argue that AI research should set a stronger focus on learning CGPMs of the hidden causes that lead to the registered observations. Endowed with suitable information-processing biases, AI may develop that will be able to explain the reality it is confronted with, reason about it, and find adaptive solutions, making it Strong AI. Seeing that such Strong AI can be equipped with a mental capacity and computational resources that exceed those of humans, the resulting system may have the potential to guide our knowledge, technology, and policies into sustainable directions. Clearly, though, Strong AI may also be used to manipulate us even more. Thus, it will be on us to put good, far-reaching and long-term, homeostasis-oriented purpose into these machines.

Event-Predictive Cognition: Underspecification and Interaction With Language

Warren and Dresang comment on the contributions from a psycholinguistic perspective, highlighting close relations between the respective research on events and proposing that, for example, verbs may indeed directly pre‐activate templates of the typically involved event participants.

Tea With Milk? A Hierarchical Generative Framework of Sequential Event Comprehension

To make sense of the world around us, we must be able to segment a continual stream of sensory inputs into discrete events. In this review, I propose that in order to comprehend events, we engage hierarchical generative models that "reverse engineer" the intentions of other agents as they produce sequential action in real time. By generating probabilistic predictions for upcoming events, generative models ensure that we are able to keep up with the rapid pace at which perceptual inputs unfold. By tracking our certainty about other agents' goals and the magnitude of prediction errors at multiple temporal scales, generative models enable us to detect event boundaries by inferring when a goal has changed. Moreover, by adapting flexibly to the broader dynamics of the environment and our own comprehension goals, generative models allow us to optimally allocate limited resources. Finally, I argue that we use generative models not only to comprehend events but also to produce events (carry out goal-relevant sequential action) and to continually learn about new events from our surroundings. Taken together, this hierarchical generative framework provides new insights into how the human brain processes events so effortlessly while highlighting the fundamental links between event comprehension, production, and learning.

Event-Predictive Cognition: A Root for Conceptual Human Thought

Our minds navigate a continuous stream of sensorimotor experiences, selectively compressing them into events. Event-predictive encodings and processing abilities have evolved because they mirror interactions between agents and objects-and the pursuance or avoidance of critical interactions lies at the heart of survival and reproduction. However, it appears that these abilities have evolved not only to pursue live-enhancing events and to avoid threatening events, but also to distinguish food sources, to produce and to use tools, to cooperate, and to communicate. They may have even set the stage for the formation of larger societies and the development of cultural identities. Research on event-predictive cognition investigates how events and conceptualizations thereof are learned, structured, and processed dynamically. It suggests that event-predictive encodings and processes optimally mediate between sensorimotor processes and language. On the one hand, they enable us to perceive and control physical interactions with our world in a highly adaptive, versatile, goal-directed manner. On the other hand, they allow us to coordinate complex social interactions and, in particular, to comprehend and produce language. Event-predictive learning segments sensorimotor experiences into event-predictive encodings. Once first encodings are formed, the mind learns progressively higher order compositional structures, which allow reflecting on the past, reasoning, and planning on multiple levels of abstraction. We conclude that human conceptual thought may be grounded in the principles of event-predictive cognition constituting its root.

Events, Event Prediction, and Predictive Processing

Events and event prediction are pivotal concepts across much of cognitive science, as demonstrated by the papers in this special issue. We first discuss how the study of events and the predictive processing framework may fruitfully inform each other. We then briefly point to some links to broader philosophical questions about events.