Detecting neural state transitions underlying event segmentation

A core set of neural states underlying memory reactivation of naturalistic events in the posterior medial cortex

In the posterior midline default mode network, spatial activity patterns similar to those during the initial experience are reactivated during the successful recall of past events. Prior studies have shown that these event-specific activity patterns are consistent across individuals recalling a shared experience, suggesting that common functional responses underlying episodic recall do exist. However, the spatial organization of function during episodic encoding and subsequent recall, especially in the absence of external stimuli, remains poorly understood. To address this, we leverage fMRI data collected during the encoding and recall of naturalistic movies to identify a core set of neural states in the posterior medial cortex. These states are stimulus-locked, reactivated during recall, and have a shared spatial organization across brains (i.e., individuals). We show that a surprisingly small number of these states (16 states across hemispheres) is sufficient to achieve the same levels of reactivation in the posterior medial cortex as when using the standard methods of the field. Additionally, these states are linked to actions and social-affective features of events in the movies. Our findings elucidate the properties of a common, spatially organized code within the posterior default mode network which appears during natural recollection of memories.

Aberrant neural event segmentation during a continuous social narrative in trauma-exposed older adolescents and young adults

Post-traumatic stress and major depressive disorders are associated with "overgeneral" autobiographical memory, or impaired recall of specific life events. Interpersonal trauma exposure, a risk factor for both conditions, may influence how symptomatic trauma-exposed (TE) individuals segment everyday events. The ability to parse experience into units (event segmentation) supports memory. Neural state transitions occur within a cortical hierarchy and play a key role in event segmentation, with regions like the occipital cortex, angular gyrus, and striatum involved in parsing event structure. We examined whether interpersonal trauma exposure was associated with alterations in the cortical hierarchy and striatal activity at neural state transitions in symptomatic TE versus healthy control (HC) individuals. Fifty older adolescents and young adults (29 TE, 21 HC) viewed the film "Partly Cloudy" during functional magnetic resonance imaging. A greedy-state boundary search algorithm assessed the optimal number of events, quality, and segmentation agreement of neural state transitions in the occipital cortex and angular gyrus. Striatal (nucleus accumbens, caudate, and putamen) activity was assessed at occipital and angular gyrus-evoked state transitions. Compared to HCs, TE participants displayed less occipital and greater angular gyrus-evoked optimal number of neural state transitions. TE participants also displayed lower quality of neural state segmentation solutions in occipital and angular cortices compared to HCs. Additionally, TE participants had less putamen activity at angular gyrus-evoked state transitions than HCs. This investigation provides neurobiological insights into aberrant event segmentation in symptomatic TE individuals, shedding light on mechanisms influencing overgeneral memory in trauma-related disorders.

Cortex-wide characterization of decision-making neural dynamics during spatial navigation

Decision-making during freely moving behaviors involves complex interactions among many cortical and subcortical regions. However, the spatiotemporal coordination across regions to generate a decision is less understood. Using a head-mounted widefield microscope, cortex-wide calcium dynamics were recorded in mice expressing GCaMP7f as they navigated an 8-maze using two paradigms. The first was an alternating pattern that required short term memory of the previous trial to make the correct decision and the second after a rule change to a fixed path in which rewards were delivered only on the left side. Identification of cortex-wide activation states revealed differences between the two paradigms. There was a higher probability for a visual/retrosplenial cortical state during the alternating paradigm and higher probability of a secondary motor and posterior parietal state during left-only. Three state sequences (motifs) illustrated both anterior and posterior activity propagations across the cortex. The anterior propagating motifs had the highest probability around the decision and posterior propagating motifs peaked following the decision. The latter, likely reflecting internal feedback to influence future actions, were more common in the left-only paradigm. Therefore, the probabilities and sequences of cortical states differ when working memory is required versus a fixed trajectory reward paradigm.

The Relationship between Event Boundary Strength and Pattern Shifts across the Cortical Hierarchy during Naturalistic Movie-viewing

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: Phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multivoxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-participant agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile in both group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual participant level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. In addition, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Predicting subjective ratings of affect and comprehensibility with text features: a reader response study of narrative poetry

Literary reading is an interactive process between a reader and a text that depends on a balance between cognitive effort and emotional rewards. By studying both the crucial features of the text and of the subjective reader reception, a better understanding of this interactive process can be reached. In the present study, subjects (N=31) read and rated a work of narrative fiction that was written in a poetic style, thereby offering the readers two pathways to cognitive rewards: Aesthetic appreciation and narrative immersion. Using purely text-based quantitative descriptors, we were able to independently and accurately predict the subjective ratings in the dimensions comprehensibility, valence, arousal, and liking across roughly 140 pages of naturalistic text. The specific text features that were most important in predicting each rating dimension are discussed in detail. In addition, the implications of the findings are discussed more generally in the context of existing models of literary processing and future research avenues for empirical literary studies.

The relationship between event boundary strength and pattern shifts across the cortical hierarchy during naturalistic movie-viewing

Our continuous experience is spontaneously segmented by the brain into discrete events. However, the beginning of a new event (an event boundary) is not always sharply identifiable: phenomenologically, event boundaries vary in salience. How are the response profiles of cortical areas at event boundaries modulated by boundary strength during complex, naturalistic movie-viewing? Do cortical responses scale in a graded manner with boundary strength, or do they merely detect boundaries in a binary fashion? We measured "cortical boundary shifts" as transient changes in multi-voxel patterns at event boundaries with different strengths (weak, moderate, and strong), determined by across-subject agreement. Cortical regions with different processing timescales were examined. In auditory areas, which have short timescales, cortical boundary shifts exhibited a clearly graded profile both in group-level and individual-level analyses. In cortical areas with long timescales, including the default mode network, boundary strength modulated pattern shift magnitude at the individual subject level. We also observed a positive relationship between boundary strength and the extent of temporal alignment of boundary shifts across different levels of the cortical hierarchy. Additionally, hippocampal activity was highest at event boundaries for which cortical boundary shifts were most aligned across hierarchical levels. Overall, we found that event boundary strength modulated cortical pattern shifts strongly in sensory areas and more weakly in higher-level areas, and that stronger boundaries were associated with greater alignment of these shifts across the cortical hierarchy.

Working memory capacity for continuous events: The root of temporal compression in episodic memory?

Remembering the unfolding of past episodes usually takes less time than their actual duration. In this study, we evaluated whether such temporal compression emerges when continuous events are too long to be fully held in working memory. To do so, we asked 90 young adults to watch and mentally replay video clips showing people performing a continuous action (e.g., turning a car jack) that lasted 3, 6, 9, 12, or 15 s. For each clip, participants had to carefully watch the event and then to mentally replay it as accurately and precisely as possible. Results showed that mental replay durations increased with event duration but in a non-linear manner: they were close to the actual event duration for short videos (3-9 s), but significantly smaller for longer videos (12 and 15 s). These results suggest that working memory is temporally limited in its capacity to represent continuous events, which could in part explain why the unfolding of events is temporally compressed in episodic memory.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

More than a moment: What does it mean to call something an 'event'?

Experiences are stored in the mind as discrete mental units, or 'events,' which influence-and are influenced by-attention, learning, and memory. In this way, the notion of an 'event' is foundational to cognitive science. However, despite tremendous progress in understanding the behavioral and neural signatures of events, there is no agreed-upon definition of an event. Here, we discuss different theoretical frameworks of event perception and memory, noting what they can and cannot account for in the literature. We then highlight key aspects of events that we believe should be accounted for in theories of event processing--in particular, we argue that the structure and substance of events should be better reflected in our theories and paradigms. Finally, we discuss empirical gaps in the event cognition literature and what the future of event cognition research may look like.

Aberrant neural processing of event boundaries in persons with Parkinson's disease

The perception of everyday events implies the segmentation into discrete sub-events (i.e. event segmentation). This process is relevant for the prediction of upcoming events and for the recall of recent activities. It is thought to involve dopaminergic networks which are strongly compromised in Parkinson's disease (PD). Indeed, deficits of event segmentation have been previously shown in PD, but underlying neuronal mechanisms remain unknown. We therefore investigated 22 persons with PD and 22 age-matched healthy controls, who performed an event segmentation task with simultaneous electroencephalography (EEG). Both groups had to indicate by button press the beginning of sub-events within three movies showing persons performing everyday activities. The segmentation performance of persons with PD deviated significantly from that of controls. Neurophysiologically, persons with PD expressed reduced theta (4-7 Hz) activity around identified event boundaries compared to healthy controls. Together, these results point to disturbed event processing in PD. According to functions attributed to EEG activities in particular frequency ranges, the PD-related theta reduction could reflect impaired matching of perceptual input with stored event representations and decreased updating processes of event information in working memory and, thus, event boundary identification.

Distinct mesoscale cortical dynamics encode search strategies during spatial navigation

Spatial navigation is a complex cognitive process that involves neural computations in distributed regions of the brain. Little is known about how cortical regions are coordinated when animals navigate novel spatial environments or how that coordination changes as environments become familiar. We recorded mesoscale calcium (Ca2+) dynamics across large swathes of the dorsal cortex in mice solving the Barnes maze, a 2D spatial navigation task where mice used random, serial, and spatial search strategies to navigate to the goal. Cortical dynamics exhibited patterns of repeated calcium activity with rapid and abrupt shifts between cortical activation patterns at sub-second time scales. We used a clustering algorithm to decompose the spatial patterns of cortical calcium activity in a low dimensional state space, identifying 7 states, each corresponding to a distinct spatial pattern of cortical activation, sufficient to describe the cortical dynamics across all the mice. When mice used serial or spatial search strategies to navigate to the goal, the frontal regions of the cortex were reliably activated for prolonged durations of time (> 1s) shortly after trial initiation. These frontal cortex activation events coincided with mice approaching the edge of the maze from the center and were preceded by temporal sequences of cortical activation patterns that were distinct for serial and spatial search strategies. In serial search trials, frontal cortex activation events were preceded by activation of the posterior regions of the cortex followed by lateral activation of one hemisphere. In spatial search trials, frontal cortical events were preceded by activation of posterior regions of the cortex followed by broad activation of the lateral regions of the cortex. Our results delineated cortical components that differentiate goal- and non-goal oriented spatial navigation strategies.

Evidence That Event Boundaries Are Access Points for Memory Retrieval

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

A Computational Complexity Perspective on Segmentation as a Cognitive Subcomputation

Computational feasibility is a widespread concern that guides the framing and modeling of natural and artificial intelligence. The specification of cognitive system capacities is often shaped by unexamined intuitive assumptions about the search space and complexity of a subcomputation. However, a mistaken intuition might make such initial conceptualizations misleading for what empirical questions appear relevant later on. We undertake here computational-level modeling and complexity analyses of segmentation - a widely hypothesized subcomputation that plays a requisite role in explanations of capacities across domains, such as speech recognition, music cognition, active sensing, event memory, action parsing, and statistical learning - as a case study to show how crucial it is to formally assess these assumptions. We mathematically prove two sets of results regarding computational hardness and search space size that may run counter to intuition, and position their implications with respect to existing views on the subcapacity.

Representational dynamics of memories for real-life events

The continuous flow of experience that characterizes real-life events is not recorded as such in episodic memory but is condensed as a succession of event segments separated by temporal discontinuities. To unravel the neural basis of this representational structure, we recorded real-life events using wearable camera technology and used fMRI to investigate brain activity during their temporal unfolding in memory. We found that, compared to the representation of static scenes in memory, dynamically unfolding memory representations were associated with greater activation of the posterior medial episodic network. Strikingly, by analyzing the autocorrelation of brain activity patterns at successive time points throughout the retrieval period, we found that this network showed higher temporal dynamics when recalling events that included a higher density of event segments. These results reveal the key role of the posterior medial network in representing the dynamic unfolding of the event segments that constitute real-world memories.

Neural event segmentation of continuous experience in human infants

How infants experience the world is fundamental to understanding their cognition and development. A key principle of adult experience is that, despite receiving continuous sensory input, we perceive this input as discrete events. Here we investigate such event segmentation in infants and how it differs from adults. Research on event cognition in infants often uses simplified tasks in which (adult) experimenters help solve the segmentation problem for infants by defining event boundaries or presenting discrete actions/vignettes. This presupposes which events are experienced by infants and leaves open questions about the principles governing infant segmentation. We take a different, data-driven approach by studying infant event segmentation of continuous input. We collected whole-brain functional MRI (fMRI) data from awake infants (and adults, for comparison) watching a cartoon and used a hidden Markov model to identify event states in the brain. We quantified the existence, timescale, and organization of multiple-event representations across brain regions. The adult brain exhibited a known hierarchical gradient of event timescales, from shorter events in early visual regions to longer events in later visual and associative regions. In contrast, the infant brain represented only longer events, even in early visual regions, with no timescale hierarchy. The boundaries defining these infant events only partially overlapped with boundaries defined from adult brain activity and behavioral judgments. These findings suggest that events are organized differently in infants, with longer timescales and more stable neural patterns, even in sensory regions. This may indicate greater temporal integration and reduced temporal precision during dynamic, naturalistic perception.

A partially nested cortical hierarchy of neural states underlies event segmentation in the human brain

A fundamental aspect of human experience is that it is segmented into discrete events. This may be underpinned by transitions between distinct neural states. Using an innovative data-driven state segmentation method, we investigate how neural states are organized across the cortical hierarchy and where in the cortex neural state boundaries and perceived event boundaries overlap. Our results show that neural state boundaries are organized in a temporal cortical hierarchy, with short states in primary sensory regions, and long states in lateral and medial prefrontal cortex. State boundaries are shared within and between groups of brain regions that resemble well-known functional networks. Perceived event boundaries overlap with neural state boundaries across large parts of the cortical hierarchy, particularly when those state boundaries demarcate a strong transition or are shared between brain regions. Taken together, these findings suggest that a partially nested cortical hierarchy of neural states forms the basis of event segmentation.

Effects of spatial boundaries on episodic memory development

Children's spatial mapping starts out particularly sensitive to 3D wall-like boundaries and develops over early childhood to flexibly include other boundary types. This study investigated whether spatial boundaries influence children's episodic memory, as in adults, and whether this effect is modulated by boundary type. Eighty-one Korean children (34 girls, 36-84 months old) re-enacted a sequence of three discrete hiding events within a space containing one of three boundaries: 3D wall, aligned objects, or 2D line. Children's memory of events occurring on one side of the boundary developed earlier than those that crossed the boundary. At first, this interaction only applied to the 3D wall and extended to other boundary types with age, suggesting that children's changing spatial representations influence their episodic memory development.

High-Order Areas and Auditory Cortex Both Represent the High-Level Event Structure of Music

Recent fMRI studies of event segmentation have found that default mode regions represent high-level event structure during movie watching. In these regions, neural patterns are relatively stable during events and shift at event boundaries. Music, like narratives, contains hierarchical event structure (e.g., sections are composed of phrases). Here, we tested the hypothesis that brain activity patterns in default mode regions reflect the high-level event structure of music. We used fMRI to record brain activity from 25 participants (male and female) as they listened to a continuous playlist of 16 musical excerpts and additionally collected annotations for these excerpts by asking a separate group of participants to mark when meaningful changes occurred in each one. We then identified temporal boundaries between stable patterns of brain activity using a hidden Markov model and compared the location of the model boundaries to the location of the human annotations. We identified multiple brain regions with significant matches to the observer-identified boundaries, including auditory cortex, medial prefrontal cortex, parietal cortex, and angular gyrus. From these results, we conclude that both higher-order and sensory areas contain information relating to the high-level event structure of music. Moreover, the higher-order areas in this study overlap with areas found in previous studies of event perception in movies and audio narratives, including regions in the default mode network.

Narrative event segmentation in the cortical reservoir

Recent research has revealed that during continuous perception of movies or stories, humans display cortical activity patterns that reveal hierarchical segmentation of event structure. Thus, sensory areas like auditory cortex display high frequency segmentation related to the stimulus, while semantic areas like posterior middle cortex display a lower frequency segmentation related to transitions between events. These hierarchical levels of segmentation are associated with different time constants for processing. Likewise, when two groups of participants heard the same sentence in a narrative, preceded by different contexts, neural responses for the groups were initially different and then gradually aligned. The time constant for alignment followed the segmentation hierarchy: sensory cortices aligned most quickly, followed by mid-level regions, while some higher-order cortical regions took more than 10 seconds to align. These hierarchical segmentation phenomena can be considered in the context of processing related to comprehension. In a recently described model of discourse comprehension word meanings are modeled by a language model pre-trained on a billion word corpus. During discourse comprehension, word meanings are continuously integrated in a recurrent cortical network. The model demonstrates novel discourse and inference processing, in part because of two fundamental characteristics: real-world event semantics are represented in the word embeddings, and these are integrated in a reservoir network which has an inherent gradient of functional time constants due to the recurrent connections. Here we demonstrate how this model displays hierarchical narrative event segmentation properties beyond the embeddings alone, or their linear integration. The reservoir produces activation patterns that are segmented by a hidden Markov model (HMM) in a manner that is comparable to that of humans. Context construction displays a continuum of time constants across reservoir neuron subsets, while context forgetting has a fixed time constant across these subsets. Importantly, virtual areas formed by subgroups of reservoir neurons with faster time constants segmented with shorter events, while those with longer time constants preferred longer events. This neurocomputational recurrent neural network simulates narrative event processing as revealed by the fMRI event segmentation algorithm provides a novel explanation of the asymmetry in narrative forgetting and construction. The model extends the characterization of online integration processes in discourse to more extended narrative, and demonstrates how reservoir computing provides a useful model of cortical processing of narrative structure.