A partially nested cortical hierarchy of neural states underlies event segmentation in the human brain

The Relationship between Event Boundary Strength and Pattern Shifts across the Cortical Hierarchy during Naturalistic Movie-viewing

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: Phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multivoxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-participant agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile in both group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual participant level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. In addition, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Causal and Chronological Relationships Predict Memory Organization for Nonlinear Narratives

While recounting an experience, one can employ multiple strategies to transition from one part to the next. For instance, if the event was learned out of linear order, one can recall events according to the time they were learned (temporal), similar events (semantic), events occurring nearby in time (chronological), or events produced by the current event (causal). To disentangle the importance of these factors, we had participants watch the nonlinear narrative, Memento, under different task instructions and presentation orders. For each scene of the film, we also separately computed semantic and causal networks. We then contrasted the evidence for temporal, semantic, chronological, or causal strategies during recall. Critically, there was stronger evidence for the causal and chronological strategies than semantic or temporal strategies. Moreover, the causal and chronological strategies outperformed the temporal one even when we asked participants to recall the film in the presented order, underscoring the fundamental nature of causal structure in scaffolding understanding and organizing recall. Nevertheless, time still marginally predicted recall transitions, suggesting it operates as a weak signal in the presence of more salient forms of structure. In addition, semantic and causal network properties predicted scene memorability, including a stronger role for incoming causes to an event than its outgoing effects. In summary, these findings highlight the importance of accounting for complex, causal networks in knowledge building and memory.

Top-down attention shifts behavioral and neural event boundaries in narratives with overlapping event scripts

Understanding and remembering the complex experiences of everyday life relies critically on prior schematic knowledge about how events in our world unfold over time. How does the brain construct event representations from a library of schematic scripts, and how does activating a specific script impact the way that events are segmented in time? We developed a novel set of 16 audio narratives, each of which combines one of four location-relevant event scripts (restaurant, airport, grocery store, and lecture hall) with one of four socially relevant event scripts (breakup, proposal, business deal, and meet cute), and presented them to participants in an fMRI study and a separate online study. Responses in the angular gyrus, parahippocampal gyrus, and subregions of the medial prefrontal cortex (mPFC) were driven by scripts related to both location and social information, showing that these regions can track schematic sequences from multiple domains. For some stories, participants were primed to attend to one of the two scripts by training them to listen for and remember specific script-relevant episodic details. Activating a location-related event script shifted the timing of subjective event boundaries to align with script-relevant changes in the narratives, and this behavioral shift was mirrored in the timing of neural responses, with mPFC event boundaries (identified using a hidden Markov model) aligning to location-relevant rather than socially relevant boundaries when participants were location primed. Our findings demonstrate that neural event dynamics are actively modulated by top-down goals and provide new insight into how narrative event representations are constructed through the activation of temporally structured prior knowledge.

Behavioural interference at event boundaries reduces long-term memory performance in the virtual water maze task without affecting working memory performance

Narrative episodic memory of movie clips can be retroactively impaired by presenting unrelated stimuli coinciding with event boundaries. This effect has been linked with rapid hippocampal processes triggered by the offset of the event, that are alternatively related either to memory consolidation or with working memory processes. Here we tested whether this effect extended to spatial memory, the temporal specificity and extent of the interference, and its effect on working- vs long-term memory. In three computerized adaptations of the Morris Water Maze, participants learned the location of an invisible target over three trials each. A second spatial navigation task was presented either immediately after finding the target, after a 10-s delay, or no second task was presented (control condition). A recall session, in which participants indicated the learned target location with 10 'pin-drop' trials for each condition, was performed after a 1-h or a 24-h break. Spatial memory was measured by the mean distance between pins and the true location. Results indicated that the immediate presentation of the second task led to worse memory performance, for both break durations, compared to the delayed condition. There was no difference in performance between the delayed presentation and the control condition. Despite this long-term memory effect, we found no difference in the rate of performance improvement during the learning session, indicating no effect of the second task on working memory. Our findings are in line with a rapid process, linked to the offset of an event, that is involved in the early stages of memory consolidation.

The metacontrol of event segmentation-A neurophysiological and behavioral perspective

During our everyday life, the constant flow of information is divided into discrete events, a process conceptualized in Event Segmentation Theory (EST). How people perform event segmentation and the resulting granularity of encapsulated segments likely depends on their metacontrol style. Yet, the underlying neural mechanisms remain undetermined. The current study examines how the metacontrol style affects event segmentation through the analysis of EEG data using multivariate pattern analysis (MVPA) and source localization analysis. We instructed two groups of healthy participants to either segment a movie as fine-grained as possible (fine-grain group) or provided no such instruction (free-segmentation group). The fine-grain group showed more segments and a higher likelihood to set event boundaries upon scene changes, which supports the notion that cognitive control influences segmentation granularity. On a neural level, representational dynamics were decodable 400 ms prior to the decision to close a segment and open a new one, and especially fronto-polar regions (BA10) were associated with this representational dynamic. Groups differed in their use of this representational dynamics to guide behavior and there was a higher sensitivity to incoming information in the Fine-grain group. Moreover, a higher likelihood to set event boundaries was reflected by activity increases in the insular cortex suggesting an increased monitoring of potentially relevant upcoming events. The study connects the EST with the metacontrol framework and relates these to overarching neural concepts of prefrontal cortex function.

The relationship between event boundary strength and pattern shifts across the cortical hierarchy during naturalistic movie-viewing

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multi-voxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-subject agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile both in group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual subject level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. Additionally, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas, and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Taking time to compose thoughts with prefrontal schemata

Under what conditions can prefrontal cortex direct the composition of brain states, to generate coherent streams of thoughts? Using a simplified Potts model of cortical dynamics, crudely differentiated into two halves, we show that once activity levels are regulated, so as to disambiguate a single temporal sequence, whether the contents of the sequence are mainly determined by the frontal or by the posterior half, or by neither, depends on statistical parameters that describe its microcircuits. The frontal cortex tends to lead if it has more local attractors, longer lasting and stronger ones, in order of increasing importance. Its guidance is particularly effective to the extent that posterior cortices do not tend to transition from state to state on their own. The result may be related to prefrontal cortex enforcing its temporally-oriented schemata driving coherent sequences of brain states, unlike the atemporal "context" contributed by the hippocampus. Modelling a mild prefrontal (vs. posterior) lesion offers an account of mind-wandering and event construction deficits observed in prefrontal patients.

The 'L-factor': Language as a transdiagnostic dimension in psychopathology

Thoughts and moods constituting our mental life incessantly change. When the steady flow of this dynamics diverges in clinical directions, the possible pathways involved are captured through discrete diagnostic labels. Yet a single vulnerable neurocognitive system may be causally involved in psychopathological deviations transdiagnostically. We argue that language viewed as integrating cortical functions is the best current candidate, whose forms of breakdown along its different dimensions are then manifest as symptoms - from prosodic abnormalities and rumination in depression to distortions of speech perception in verbal hallucinations, distortions of meaning and content in delusions, or disorganized speech in formal thought disorder. Spontaneous connected speech provides continuous objective readouts generating a highly accessible bio-behavioral marker with the potential of revolutionizing neuropsychological measurement. This argument turns language into a transdiagnostic 'L-factor' providing an analytical and mechanistic substrate for previously proposed latent general factors of psychopathology ('p-factor') and cognitive functioning ('c-factor'). Together with immense practical opportunities afforded by rapidly advancing natural language processing (NLP) technologies and abundantly available data, this suggests a new era of translational clinical psychiatry, in which both psychopathology and language may be rethought together.

Contemporary neurocognitive models of memory: A descriptive comparative analysis

The great complexity involved in the study of memory has given rise to numerous hypotheses and models associated with various phenomena at different levels of analysis. This has allowed us to delve deeper in our knowledge about memory but has also made it difficult to synthesize and integrate data from different lines of research. In this context, this work presents a descriptive comparative analysis of contemporary models that address the structure and function of multiple memory systems. The main goal is to outline a panoramic view of the key elements that constitute these models in order to visualize both the current state of research and possible future directions. The elements that stand out from different levels of analysis are distributed neural networks, hierarchical organization, predictive coding, homeostasis, and evolutionary perspective.

Toward an integrative account of internal and external determinants of event segmentation

Our daily experiences unfold continuously, but we remember them as a series of discrete events through a process called event segmentation. Prominent theories of event segmentation suggest that event boundaries in memory are triggered by significant shifts in the external environment, such as a change in one's physical surroundings. In this review, we argue for a fundamental extension of this research field to also encompass internal state changes as playing a key role in structuring event memory. Accordingly, we propose an expanded taxonomy of event boundary-triggering processes, and review behavioral and neuroscience research on internal state changes in three core domains: affective states, goal states, and motivational states. Finally, we evaluate how well current theoretical frameworks can accommodate the unique and interactive contributions of internal states to event memory. We conclude that a theoretical perspective on event memory that integrates both external environment and internal state changes allows for a more complete understanding of how the brain structures experiences, with important implications for future research in cognitive and clinical neuroscience.

More than a moment: What does it mean to call something an 'event'?

Experiences are stored in the mind as discrete mental units, or 'events,' which influence-and are influenced by-attention, learning, and memory. In this way, the notion of an 'event' is foundational to cognitive science. However, despite tremendous progress in understanding the behavioral and neural signatures of events, there is no agreed-upon definition of an event. Here, we discuss different theoretical frameworks of event perception and memory, noting what they can and cannot account for in the literature. We then highlight key aspects of events that we believe should be accounted for in theories of event processing--in particular, we argue that the structure and substance of events should be better reflected in our theories and paradigms. Finally, we discuss empirical gaps in the event cognition literature and what the future of event cognition research may look like.

Setting boundaries: Development of neural and behavioral event cognition in early childhood

The ongoing stream of sensory experience is so complex and ever-changing that we tend to parse this experience at "event boundaries," which structures and strengthens memory. Memory processes undergo profound change across early childhood. Whether young children also divide their ongoing processing along event boundaries, and if those boundaries relate to memory, could provide important insight into the development of memory systems. In Study 1, 4-7-year-old children and adults segmented a cartoon, and we tested their memory. Children's event boundaries were more variable than adults' and differed in location and consistency of agreement. Older children's event segmentation was more adult-like than younger children's, and children who segmented events more like adults had better memory for those events. In Study 2, we asked whether these developmental differences in event segmentation had their roots in distinct neural representations. A separate group of 4-8-year-old children watched the same cartoon while undergoing an fMRI scan. In the right hippocampus, greater pattern dissimilarity across event boundaries compared to within events was evident for both child and adult behavioral boundaries, suggesting children and adults share similar event cognition. However, the boundaries identified by a data-driven Hidden Markov Model found that a different brain region-the left and right angular gyrus-aligned only with event boundaries defined by children. Overall, these data suggest that children's event cognition is reasonably well-developed by age 4 but continues to become more adult-like across early childhood. RESEARCH HIGHLIGHTS: Adults naturally break their experience into events, which structures and strengthens memory, but less is known about children's event perception and memory. Study 1 had adults and children segment and remember events from an animated show, and Study 2 compared those segmentations to other children's fMRI data. Children show better recognition and temporal order memory and more adult-like event segmentation with age, and children who segment more like adults have better memory. Children's and adults' behavioral boundaries mapped onto pattern similarity differences in hippocampus, and children's behavioral boundaries matched a data-driven model's boundaries in angular gyrus.

Opposite effects of emotion and event segmentation on temporal order memory and object-context binding

Our daily lives unfold continuously, yet our memories are organised into distinct events, situated in a specific context of space and time, and chunked when this context changes (at event boundaries). Previous research showed that this process, termed event segmentation, enhances object-context binding but impairs temporal order memory. Physiologically, peaks in pupil dilation index event segmentation, similar to emotion-induced bursts of autonomic arousal. Emotional arousal also modulates object-context binding and temporal order memory. Yet, these two critical factors have not been systematically studied together. To address this gap, we ran a behavioural experiment using a paradigm validated to study event segmentation and extended it with emotion manipulation. During encoding, we sequentially presented greyscale objects embedded in coloured frames (colour changes defining events), with a neutral or aversive sound. During retrieval, we tested participants' memory of temporal order memory and object-colour binding. We found opposite effects of emotion and event segmentation on episodic memory. While event segmentation enhanced object-context binding, emotion impaired it. On the contrary, event segmentation impaired temporal order memory, but emotion enhanced it. These findings increase our understanding of episodic memory organisation in laboratory settings, and potentially in real life with perceptual changes and emotion fluctuations constantly interacting.

20 years of the default mode network: A review and synthesis

The discovery of the default mode network (DMN) has revolutionized our understanding of the workings of the human brain. Here, I review developments that led to the discovery of the DMN, offer a personal reflection, and consider how our ideas of DMN function have evolved over the past two decades. I summarize literature examining the role of the DMN in self-reference, social cognition, episodic and autobiographical memory, language and semantic memory, and mind wandering. I identify unifying themes and propose new perspectives on the DMN's role in human cognition. I argue that the DMN integrates and broadcasts memory, language, and semantic representations to create a coherent "internal narrative" reflecting our individual experiences. This narrative is central to the construction of a sense of self, shapes how we perceive ourselves and interact with others, may have ontogenetic origins in self-directed speech during childhood, and forms a vital component of human consciousness.

Evidence That Event Boundaries Are Access Points for Memory Retrieval

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

Information flow across the cortical timescale hierarchy during narrative construction

When listening to spoken narratives, we must integrate information over multiple, concurrent timescales, building up from words to sentences to paragraphs to a coherent narrative. Recent evidence suggests that the brain relies on a chain of hierarchically organized areas with increasing temporal receptive windows to process naturalistic narratives. We hypothesized that the structure of this cortical processing hierarchy should result in an observable sequence of response lags between networks comprising the hierarchy during narrative comprehension. This study uses functional MRI to estimate the response lags between functional networks during narrative comprehension. We use intersubject cross-correlation analysis to capture network connectivity driven by the shared stimulus. We found a fixed temporal sequence of response lags-on the scale of several seconds-starting in early auditory areas, followed by language areas, the attention network, and lastly the default mode network. This gradient is consistent across eight distinct stories but absent in data acquired during rest or using a scrambled story stimulus, supporting our hypothesis that narrative construction gives rise to internetwork lags. Finally, we build a simple computational model for the neural dynamics underlying the construction of nested narrative features. Our simulations illustrate how the gradual accumulation of information within the boundaries of nested linguistic events, accompanied by increased activity at each level of the processing hierarchy, can give rise to the observed lag gradient.