Neural pattern change during encoding of a narrative predicts retrospective duration estimates

The influence of emotion on temporal context models

Temporal context models (TCMs) have been influential in understanding episodic memory and its neural underpinnings. Recently, TCMs have been extended to explain emotional memory effects, one of the most clinically important findings in the field of memory research. This review covers recent advances in hypotheses for the neural representation of spatiotemporal context through the lens of TCMs, including their ability to explain the influence of emotion on episodic and temporal memory. In recent years, simplifying assumptions of "classical" TCMs - with exponential trace decay and the mechanism by which temporal context is recovered - have become increasingly clear. The review also outlines how recent advances could be incorporated into a future TCM, beyond classical assumptions, to integrate emotional modulation.

Toward an integrative account of internal and external determinants of event segmentation

Our daily experiences unfold continuously, but we remember them as a series of discrete events through a process called event segmentation. Prominent theories of event segmentation suggest that event boundaries in memory are triggered by significant shifts in the external environment, such as a change in one's physical surroundings. In this review, we argue for a fundamental extension of this research field to also encompass internal state changes as playing a key role in structuring event memory. Accordingly, we propose an expanded taxonomy of event boundary-triggering processes, and review behavioral and neuroscience research on internal state changes in three core domains: affective states, goal states, and motivational states. Finally, we evaluate how well current theoretical frameworks can accommodate the unique and interactive contributions of internal states to event memory. We conclude that a theoretical perspective on event memory that integrates both external environment and internal state changes allows for a more complete understanding of how the brain structures experiences, with important implications for future research in cognitive and clinical neuroscience.

Asynchronous development of memory integration and differentiation influences temporal memory organization

Adults remember items with shared contexts as occurring closer in time to one another than those associated with different contexts, even when their objective temporal distance is fixed. Such temporal memory biases are thought to reflect within-event integration and between-event differentiation processes that organize events according to their contextual similarities and differences, respectively. Within-event integration and between-event differentiation are hypothesized to differentially rely on binding and control processes, which may develop at different ages. To test this hypothesis, 5- to 12-year-olds and adults (N = 134) studied quartets of image pairs that contained either the same scene (same-context) or different scenes (different-context). Participants remembered same-context items as occurring closer in time by older childhood (7-9 years), whereas different-context items were remembered as occurring farther apart by early adolescence (10-12 years). The differential emergence of these temporal memory biases suggests within-event integration and between-event differentiation emerge at different ages. RESEARCH HIGHLIGHTS: Children are less likely than adults to use contextual information (e.g., location) to organize their continuous experience in memory, as indicated by temporal memory biases. Biases reflecting within-event integration (i.e., remembering elements with a shared context as occurring closer together in time) emerged in late childhood. Biases reflecting between-event differentiation (i.e., remembering elements from different contexts as occurring farther apart in time) emerged in early adolescence. The differential emergence of biases reflecting within-event integration and between-event differentiation suggests they are distinct, yet complementary, processes that support developmental improvements in event memory organization.

Prediction error and event segmentation in episodic memory

Organizing the continuous flow of experiences into meaningful events is a crucial prerequisite for episodic memory. Prediction error and event segmentation both play important roles in supporting the genesis of meaningful mnemonic representations of events. We review theoretical contributions discussing the relationship between prediction error and event segmentation, as well as literature on episodic memory related to prediction error and event segmentation. We discuss the extent of overlap of mechanisms underlying memory emergence through prediction error and event segmentation, with a specific focus on attention and working memory. Finally, we identify areas in research that are currently developing and suggest future directions. We provide an overview of mechanisms underlying memory formation through predictions, violations of predictions, and event segmentation.

Investigating the impact of healthy aging on memory for temporal duration and order

Temporal information, including information about temporal order and duration, is a fundamental component of event sequence memory. While previous research has demonstrated that aging can have a detrimental effect on memory for temporal order, there has been limited insight into the effect of aging on memory for durations, particularly within the context of sequences. In the current study, neurologically healthy young and older participants were administered two temporal match-mismatch tasks: one in which they were instructed on each trial to compare the temporal order or duration information of stimulus sequences presented first in a study phase and then, after a short delay, in a test phase (event sequence task); and a second in which participants were required to compare single durations or sequences of durations across study and test phases of each trial (pinwheel task). Consistent with the literature, the older participants were significantly poorer compared to their younger counterparts at making temporal order match-mismatch judgments in the event sequence task. In addition to this, data from both tasks suggested that the older adults were also less accurate at match-mismatch judgments based on duration information, with tentative evidence from the pinwheel task to suggest that this age-related effect was most prominent when the duration information was presented within a sequence. We suggest that age-related changes to medial temporal and frontal lobe function may contribute to changes in memory for temporal information in older adults, given the importance of these regions to event sequence memory.

Minute-scale periodicity of neuronal firing in the human entorhinal cortex

Grid cells in the entorhinal cortex demonstrate spatially periodic firing, thought to provide a spatial map on behaviorally relevant length scales. Whether such periodicity exists for behaviorally relevant time scales in the human brain remains unclear. We investigate neuronal firing during a temporally continuous experience by presenting 14 neurosurgical patients with a video while recording neuronal activity from multiple brain regions. We report on neurons that modulate their activity in a periodic manner across different time scales-from seconds to many minutes, most prevalently in the entorhinal cortex. These neurons remap their dominant periodicity to shorter time scales during a subsequent recognition memory task. When the video is presented at two different speeds, a significant percentage of these temporally periodic cells (TPCs) maintain their time scales, suggesting a degree of invariance. The TPCs' temporal periodicity might complement the spatial periodicity of grid cells and together provide scalable spatiotemporal metrics for human experience.

Time for Memories

The ability to store information about the past to dynamically predict and prepare for the future is among the most fundamental tasks the brain performs. To date, the problems of understanding how the brain stores and organizes information about the past (memory) and how the brain represents and processes temporal information for adaptive behavior have generally been studied as distinct cognitive functions. This Symposium explores the inherent link between memory and temporal cognition, as well as the potential shared neural mechanisms between them. We suggest that working memory and implicit timing are interconnected and may share overlapping neural mechanisms. Additionally, we explore how temporal structure is encoded in associative and episodic memory and, conversely, the influences of episodic memory on subsequent temporal anticipation and the perception of time. We suggest that neural sequences provide a general computational motif that contributes to timing and working memory, as well as the spatiotemporal coding and recall of episodes.

Time processing in neurological and psychiatric conditions

A central question in understanding cognition and pathology-related cognitive changes is how we process time. However, time processing difficulties across several neurological and psychiatric conditions remain seldom investigated. The aim of this review is to develop a unifying taxonomy of time processing, and a neuropsychological perspective on temporal difficulties. Four main temporal judgments are discussed: duration processing, simultaneity and synchrony, passage of time, and mental time travel. We present an integrated theoretical framework of timing difficulties across psychiatric and neurological conditions based on selected patient populations. This framework provides new mechanistic insights on both (a) the processes involved in each temporal judgement, and (b) temporal difficulties across pathologies. By identifying underlying transdiagnostic time-processing mechanisms, this framework opens fruitful avenues for future research.

Bayesian Surprise Predicts Human Event Segmentation in Story Listening

Event segmentation theory posits that people segment continuous experience into discrete events and that event boundaries occur when there are large transient increases in prediction error. Here, we set out to test this theory in the context of story listening, by using a deep learning language model (GPT-2) to compute the predicted probability distribution of the next word, at each point in the story. For three stories, we used the probability distributions generated by GPT-2 to compute the time series of prediction error. We also asked participants to listen to these stories while marking event boundaries. We used regression models to relate the GPT-2 measures to the human segmentation data. We found that event boundaries are associated with transient increases in Bayesian surprise but not with a simpler measure of prediction error (surprisal) that tracks, for each word in the story, how strongly that word was predicted at the previous time point. These results support the hypothesis that prediction error serves as a control mechanism governing event segmentation and point to important differences between operational definitions of prediction error.

Re-expression of CA1 and entorhinal activity patterns preserves temporal context memory at long timescales

Converging, cross-species evidence indicates that memory for time is supported by hippocampal area CA1 and entorhinal cortex. However, limited evidence characterizes how these regions preserve temporal memories over long timescales (e.g., months). At long timescales, memoranda may be encountered in multiple temporal contexts, potentially creating interference. Here, using 7T fMRI, we measured CA1 and entorhinal activity patterns as human participants viewed thousands of natural scene images distributed, and repeated, across many months. We show that memory for an image's original temporal context was predicted by the degree to which CA1/entorhinal activity patterns from the first encounter with an image were re-expressed during re-encounters occurring minutes to months later. Critically, temporal memory signals were dissociable from predictors of recognition confidence, which were carried by distinct medial temporal lobe expressions. These findings suggest that CA1 and entorhinal cortex preserve temporal memories across long timescales by coding for and reinstating temporal context information.

Neural temporal context reinstatement of event structure during memory recall

The transformation of experiences into meaningful events and memories is intertwined with the notion of time. Temporal perception can influence, and be influenced by, segmenting continuous experience into meaningful events. Episodic memories formed from these events become associated with temporal information as well. However, it is less clear how temporal perception contributes to structuring events and organizing memory: whether it plays a more active or passive role, and whether this temporal information is encoded initially during perception or influenced by retrieval processes. To address these questions, we examined how event segmentation influences temporal representations during initial perception and memory retrieval, without testing temporal information explicitly. Using a neural measure of temporal context extracted from scalp electroencephalography in human participants (N = 170), we found reduced temporal context similarity between studied items separated by an event boundary when compared to items from the same event. Furthermore, while participants freely recalled list items, neural activity reflected reinstatement of temporal context representations from the study phase, including temporal disruption. A computational model of episodic memory, the context maintenance and retrieval (CMR) model, predicted these results, and made novel predictions regarding the influence of temporal disruption on recall order. These findings implicate the impact of event structure on memory organization via temporal representations, underscoring the role of temporal information in event segmentation and episodic memory. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Evidence That Event Boundaries Are Access Points for Memory Retrieval

When recalling memories, we often scan information-rich continuous episodes, for example, to find our keys. How does our brain access and search through those memories? We suggest that high-level structure, marked by event boundaries, guides us through this process: In our computational model, memory scanning is sped up by skipping ahead to the next event boundary upon reaching a decision threshold. In adult Mechanical Turk workers from the United States, we used a movie (normed for event boundaries; Study 1, N = 203) to prompt memory scanning of movie segments for answers (Study 2, N = 298) and mental simulation (Study 3, N = 100) of these segments. Confirming model predictions, we found that memory-scanning times varied as a function of the number of event boundaries within a segment and the distance of the search target to the previous boundary (the key diagnostic parameter). Mental simulation times were also described by a skipping process with a higher skipping threshold than memory scanning. These findings identify event boundaries as access points to memory.

Mnemonic Content and Hippocampal Patterns Shape Judgments of Time

Our experience of time can feel dilated or compressed, rather than reflecting true "clock time." Although many contextual factors influence the subjective perception of time, it is unclear how memory accessibility plays a role in constructing our experience of and memory for time. Here, we used a combination of behavioral and functional MRI measures in healthy young adults (N = 147) to ask the question of how memory is incorporated into temporal duration judgments. Behaviorally, we found that event boundaries, which have been shown to disrupt ongoing memory integration processes, result in the temporal compression of duration judgments. Additionally, using a multivoxel pattern similarity analysis of functional MRI data, we found that greater temporal pattern change in the left hippocampus within individual trials was associated with longer duration judgments. Together, these data suggest that mnemonic processes play a role in constructing representations of time.

Contextual inference in learning and memory

Context is widely regarded as a major determinant of learning and memory across numerous domains, including classical and instrumental conditioning, episodic memory, economic decision-making, and motor learning. However, studies across these domains remain disconnected due to the lack of a unifying framework formalizing the concept of context and its role in learning. Here, we develop a unified vernacular allowing direct comparisons between different domains of contextual learning. This leads to a Bayesian model positing that context is unobserved and needs to be inferred. Contextual inference then controls the creation, expression, and updating of memories. This theoretical approach reveals two distinct components that underlie adaptation, proper and apparent learning, respectively referring to the creation and updating of memories versus time-varying adjustments in their expression. We review a number of extensions of the basic Bayesian model that allow it to account for increasingly complex forms of contextual learning.

Multisensory synchrony of contextual boundaries affects temporal order memory, but not encoding or recognition

We memorize our daily life experiences, which are often multisensory in nature, by segmenting them into distinct event models, in accordance with perceived contextual or situational changes. However, very little is known about how multisensory boundaries affect segmentation, as most studies have focused on unisensory (visual or audio) segmentation. In three experiments, we investigated the effect of multisensory boundaries on segmentation in memory and perception. In Experiment 1, participants encoded lists of pictures while audio and visual contexts changed synchronously or asynchronously. After each list, we tested recognition and temporal associative memory for pictures that were encoded in the same audio-visual context or that crossed a synchronous or an asynchronous multisensory change. We found no effect of multisensory synchrony for recognition memory: synchronous and asynchronous changes similarly impaired recognition for pictures encoded at those changes, compared to pictures encoded further away from those changes. Multisensory synchrony did affect temporal associative memory, which was worse for pictures encoded at synchronous than at asynchronous changes. Follow up experiments showed that this effect was not due to the higher dimensionality of multisensory over unisensory contexts (Experiment 2), nor that it was due to the temporal unpredictability of contextual changes inherent to Experiment 1 (Experiment 3). We argue that participants formed situational expectations through multisensory synchronicity, such that synchronous multisensory changes deviated more strongly from those expectations than asynchronous changes. We discuss our findings in light of supportive and conflicting findings of uni- and multi-sensory segmentation.

The tie that binds: temporal coding and adaptive emotion

Emotions are temporally dynamic, but the persistence of emotions outside of their appropriate temporal context is detrimental to health and well-being. Yet, precisely how temporal coding and emotional processing interact remains unclear. Recently unveiled temporal context representations in the hippocampus, entorhinal cortex (EC), and prefrontal cortex (PFC) support memory for what happened when. Here, we discuss how these neural temporal representations may interact with densely interconnected amygdala circuitry to shape emotional functioning. We propose a neuroanatomically informed framework suggesting that high-fidelity temporal representations linked to dynamic experiences promote emotion regulation and adaptive emotional memories. Then, we discuss how newly-identified synaptic and molecular features of amygdala-hippocampal projections suggest that intense, amygdala-dependent emotional responses may distort temporal-coding mechanisms. We conclude by identifying key avenues for future research.

The hearing hippocampus

The hippocampus has a well-established role in spatial and episodic memory but a broader function has been proposed including aspects of perception and relational processing. Neural bases of sound analysis have been described in the pathway to auditory cortex, but wider networks supporting auditory cognition are still being established. We review what is known about the role of the hippocampus in processing auditory information, and how the hippocampus itself is shaped by sound. In examining imaging, recording, and lesion studies in species from rodents to humans, we uncover a hierarchy of hippocampal responses to sound including during passive exposure, active listening, and the learning of associations between sounds and other stimuli. We describe how the hippocampus' connectivity and computational architecture allow it to track and manipulate auditory information - whether in the form of speech, music, or environmental, emotional, or phantom sounds. Functional and structural correlates of auditory experience are also identified. The extent of auditory-hippocampal interactions is consistent with the view that the hippocampus makes broad contributions to perception and cognition, beyond spatial and episodic memory. More deeply understanding these interactions may unlock applications including entraining hippocampal rhythms to support cognition, and intervening in links between hearing loss and dementia.

The neural bases for timing of durations

Durations are defined by a beginning and an end, and a major distinction is drawn between durations that start in the present and end in the future ('prospective timing') and durations that start in the past and end either in the past or the present ('retrospective timing'). Different psychological processes are thought to be engaged in each of these cases. The former is thought to engage a clock-like mechanism that accurately tracks the continuing passage of time, whereas the latter is thought to engage a reconstructive process that utilizes both temporal and non-temporal information from the memory of past events. We propose that, from a biological perspective, these two forms of duration 'estimation' are supported by computational processes that are both reliant on population state dynamics but are nevertheless distinct. Prospective timing is effectively carried out in a single step where the ongoing dynamics of population activity directly serve as the computation of duration, whereas retrospective timing is carried out in two steps: the initial generation of population state dynamics through the process of event segmentation and the subsequent computation of duration utilizing the memory of those dynamics.

Mnemonic construction and representation of temporal structure in the hippocampal formation

The hippocampal-entorhinal region supports memory for episodic details, such as temporal relations of sequential events, and mnemonic constructions combining experiences for inferential reasoning. However, it is unclear whether hippocampal event memories reflect temporal relations derived from mnemonic constructions, event order, or elapsing time, and whether these sequence representations generalize temporal relations across similar sequences. Here, participants mnemonically constructed times of events from multiple sequences using infrequent cues and their experience of passing time. After learning, event representations in the anterior hippocampus reflected temporal relations based on constructed times. Temporal relations were generalized across sequences, revealing distinct representational formats for events from the same or different sequences. Structural knowledge about time patterns, abstracted from different sequences, biased the construction of specific event times. These findings demonstrate that mnemonic construction and the generalization of relational knowledge combine in the hippocampus, consistent with the simulation of scenarios from episodic details and structural knowledge.

Activity patterns in the hippocampus resemble temporal relations of learned event sequences. Here, the authors show that these relational memories arise through mnemonic construction and are generalized to reflect the temporal event structure.

Event segmentation reveals working memory forgetting rate

We encounter the world as a continuous flow and effortlessly segment sequences of events into episodes. This process of event segmentation engages working memory (WM) for tracking the flow of events and impacts subsequent memory accuracy. WM is limited in how much information (i.e., WM capacity) and for how long the information is retained (i.e., forgetting rate). In this study, across multiple tasks, we estimated participants’ WM capacity and forgetting rate in a dynamic context and evaluated their relationship to event segmentation. A U-shaped relationship across tasks shows that individuals who segmented the movie more finely or coarsely than the average have a faster WM forgetting rate. A separate task assessing long-term memory retrieval revealed that the coarse-segmenters have better recognition of temporal order of events compared to the fine-segmenters. These findings show that event segmentation employs dissociable memory strategies and correlates with how long information is retained in WM

•

The event segmentation grain is variable across individuals

•

The event segmentation grain has a U-shaped relationship with the WM forgetting rate

•

The temporal order memory accuracy decreases with the increasing event segmentation

•

The number of recalled events increases with the increasing event segmentation

Switching task sets creates event boundaries in memory

People segregate continuously unfolding experiences into discrete events in memory. This process, known as event segmentation, results in better memory for the temporal order of experiences within an event and expands subjective temporal distance for items encoded across event boundaries. Previous research has suggested that the creation of event boundaries is driven by (typically unpredicted) changes in external stimulation, though many prior studies have confounded a change in bottom-up input with a concurrent change in task goal. This raises the question of whether event segmentation can be triggered by the endogenous cognitive control processes involved in switching task sets, independent of changes in bottom-up stimulation. We investigated this question by embedding task set changes during encoding of a series of trial-unique images, and comparing subsequent temporal order and distance memory for item pairs encoded across a change in task set with item pairs encoded within the same task set. Across five experiments, we demonstrate that both cued and voluntary task set changes are sufficient to create event boundaries, while ruling out potential confounding effects of shifts in stimulus set, response set, task cues, and task difficulty. Thus, internal control processes are a key determinant of segmenting episodic memories, and task set updating can trigger event segmentation independent of any externally induced, perceptual or task-based prediction error.

Time changes: Timing contexts support event segmentation in associative memory

We tend to mentally segment a series of events according to perceptual contextual changes, such that items from a shared context are more strongly associated in memory than items from different contexts. It is also known that timing context provides a scaffold to structure experiences in memory, but its role in event segmentation has not been investigated. We adapted a previous paradigm, which was used to investigate event segmentation using visual contexts, to study the effects of changes in timing contexts on event segmentation in associative memory. In two experiments, we presented lists of 36 items in which the interstimulus intervals (ISIs) changed after a series of six items ranging between 0.5 and 4 s in 0.5 s steps. After each list, participants judged which one of two test items were shown first (temporal order judgment) for items that were either drawn from the same context (within an ISI) or from consecutive contexts (across ISIs). Further, participants judged from memory whether the ISI associated to an item lasted longer than a standard interval (2.25 s) that was not previously shown (temporal source memory). Experiment 2 further included a time-item encoding task. Results revealed an effect of timing context changes in temporal order judgments, with faster responses (Experiment 1) or higher accuracy (Experiment 2) when items were drawn from the same context, as opposed to items drawn from across contexts. Further, in both experiments, we found that participants were well able to provide temporal source memory judgments based on recalled durations. Finally, replicated across experiments, we found subjective duration bias, as estimated by psychometric curve fitting parameters of the recalled durations, correlated negatively with within-context temporal order judgments. These findings show that changes in timing context support event segmentation in associative memory.

The "Narratives" fMRI dataset for evaluating models of naturalistic language comprehension

The "Narratives" collection aggregates a variety of functional MRI datasets collected while human subjects listened to naturalistic spoken stories. The current release includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ~4.6 hours of unique stimuli (~43,000 words). This data collection is well-suited for naturalistic neuroimaging analysis, and is intended to serve as a benchmark for models of language and narrative comprehension. We provide standardized MRI data accompanied by rich metadata, preprocessed versions of the data ready for immediate use, and the spoken story stimuli with time-stamped phoneme- and word-level transcripts. All code and data are publicly available with full provenance in keeping with current best practices in transparent and reproducible neuroimaging.

Effects of order on memory of event times

Memorizing time of an event may employ two processes (1) encoding of the absolute time of events within an episode, (2) encoding of its relative order. Here we study interaction between these two processes. We performed experiments in which one or several items were presented, after which participants were asked to report the time of occurrence of items. When a single item was presented, the distribution of reported times was quite wide. When two or three items were presented, the relative order among them strongly affected the reported time of each of them. Bayesian theory that takes into account the memory for the events order is compatible with the experimental data, in particular in terms of the effect of order on absolute time reports. Our results suggest that people do not deduce order from memorized time, instead people’s memory for absolute time of events relies critically on memorized order of the events.

Hippocampal Representations of Event Structure and Temporal Context during Episodic Temporal Order Memory

The hippocampus plays an important role in representing spatial locations and sequences and in transforming representations. How these representational structures and operations support memory for the temporal order of random items is still poorly understood. We addressed this question by leveraging the method of loci, a powerful mnemonic strategy for temporal order memory that particularly recruits hippocampus-dependent computations of spatial locations and associations. Applying representational similarity analysis to functional magnetic resonance imaging activation patterns revealed that hippocampal subfields contained representations of multiple features of sequence structure, including spatial locations, location distance, and sequence boundaries, as well as episodic-like temporal context. Critically, the hippocampal CA1 exhibited spatial transformation of representational patterns, showing lower pattern similarity for items in same locations than closely matched different locations during retrieval, whereas the CA23DG exhibited sequential transformation of representational patterns, showing lower pattern similarity for items in near locations than in far locations during encoding. These transformations enabled the encoding of multiple items in the same location and disambiguation of adjacent items. Our results suggest that the hippocampus can flexibly reconfigure multiplexed event structure representations to support accurate temporal order memory.

Hippocampal-Medial Prefrontal Event Segmentation and Integration Contribute to Episodic Memory Formation

How do we encode our continuous life experiences for later retrieval? Theories of event segmentation and integration suggest that the hippocampus binds separately represented events into an ordered narrative. Using a functional Magnetic Resonance Imaging (fMRI) movie watching-recall dataset, we quantified two types of neural similarities (i.e., "activation pattern" similarity and within-region voxel-based "connectivity pattern" similarity) between separate events during movie watching and related them to subsequent retrieval of events as well as retrieval of sequential order. We demonstrated that compared with forgotten events, successfully remembered events were associated with distinct "activation patterns" in the hippocampus and medial prefrontal cortex. In contrast, similar "connectivity pattern" between events were associated with memory formation and were also relevant for retaining events in the correct order. We applied the same approaches to an independent movie watching fMRI dataset as validation and highlighted again the role of hippocampal activation pattern and connectivity pattern in memory formation. We propose that distinct activation patterns represent neural segmentation of events, while similar connectivity patterns encode context information and, therefore, integrate events into a narrative. Our results provide novel evidence for the role of hippocampal-medial prefrontal event segmentation and integration in episodic memory formation of real-life experience.

The diminishing precision of memory for time

Knowing when an event took place can provide several benefits to episodic memory, such as distinguishing among multiple traces, learning sequences of events, and guiding a search strategy. As a tool for understanding memory, time is particularly appealing given its ever-changing quality, the constant possibility to associate it with encoded events, and the ease with which it can be targeted at retrieval. Whereas studies of episodic retrieval typically employ categorical and probabilistic measures of retrieval success, characterizing a continuous feature such as time warrants measures particularly sensitive to the fidelity, or precision, of retrieved information. Here, we adapt a paradigm for assessing the fine-grained precision of retrieval to understand the nature of judging the time at which a memory was encoded. Subjects studied a series of pictures and then undertook a test in which they placed each picture, as precisely as possible, along a continuous time line representing the study list. Based on mixture-modeling analyses of the test response errors, the primary results were that temporal judgments were less accurate with passing time, and this change was due to diminished precision as opposed to an increased rate of guessing. Moreover, although we observed a negligible influence of guessing, subjects exhibited a clear effect of bias that favored recent responses. Together, in contrast to numerous studies of memory for other continuous features (e.g., color and location), our findings demonstrate a novel pattern of decision factors, suggesting that the retrieval of time might highlight distinct attributes of episodic memory.

Prospective and retrospective timing in mild cognitive impairment and Alzheimer's disease patients: A systematic review and meta-analysis

Performance on timing tasks changes with age. Whether these changes reflect a real "clock" problem due to aging or a secondary effect of the reduced cognitive resources of older adults is still an unsettled question. Research on processing of time in aged populations marked by severe mnemonic and/or attentional deficits, such as patients with Alzheimer's disease (AD) and Mild Cognitive Impairment (MCI), may help elucidate the role of cognitive resources in age-related temporal distortions. To this end, we conducted a systematic review and meta-analysis of timing studies in AD and MCI patients; both prospective and retrospective timing tasks were considered and analysed separately. As concerns prospective timing, a first random-effect model showed a medium overall effect of neurodegeneration on timing performance. When considering the role of moderator variables(i.e., neurodegenerative condition, type of measure, participants' age and years of education, interval length, and type of timing task), mean score appeared to be a less sensitive measure than accuracy and variability, and the observed temporal impairment was smaller in older samples. In addition, AD patients only exhibited medium-to-high impairment on prospective timing tasks, whereas MCI patients did not significantly differ from controls. However, assuming a mean age of 70 years old and absolute error as dependent variable, a second fitted meta-regression model predicted a significant outcome also for MCI patients. Concerning retrospective timing, a significant but small effect of neurodegeneration was observed for retrospective judgments. None of the moderators, however, explained between-studies variability. Collectively, our findings highlight a clear deficit in prospective timing for AD patients and underscore several issues that future work should carefully consider to better investigate the effect of MCI on prospective temporal judgements. Results from retrospective timing also point to a possible impairment of retrospective judgments in neurodegenerative conditions, albeit more studies are needed to substantiate this finding.

Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience

How we process ongoing experiences is shaped by our personal history, current needs, and future goals. Consequently, ventromedial prefrontal cortex (vmPFC) activity involved in processing these subjective appraisals appears to be highly idiosyncratic across individuals. To elucidate the role of the vmPFC in processing our ongoing experiences, we developed a computational framework and analysis pipeline to characterize the spatiotemporal dynamics of individual vmPFC responses as participants viewed a 45-minute television drama. Through a combination of functional magnetic resonance imaging, facial expression tracking, and self-reported emotional experiences across four studies, our data suggest that the vmPFC slowly transitions through a series of discretized states that broadly map onto affective experiences. Although these transitions typically occur at idiosyncratic times across people, participants exhibited a marked increase in state alignment during high affectively valenced events in the show. Our work suggests that the vmPFC ascribes affective meaning to our ongoing experiences.

Reconstructive nature of temporal memory for movie scenes

Remembering when events took place is a key component of episodic memory. Using a sensitive behavioral measure, the present study investigates whether spontaneous event segmentation and script-based prior knowledge affect memory for the time of movie scenes. In three experiments, different groups of participants were asked to indicate when short video clips extracted from a previously encoded movie occurred on a horizontal timeline that represented the video duration. When participants encoded the entire movie, they were more precise at judging the temporal occurrence of clips extracted from the beginning and the end of the film compared to its middle part, but also at judging clips that were closer to event boundaries. Removing the final part of the movie from the encoding session resulted in a systematic bias in memory for time. Specifically, participants increasingly underestimated the time of occurrence of the video clips as a function of their proximity to the missing part of the movie. An additional experiment indicated that such an underestimation effect generalizes to different audio-visual material and does not necessarily reflect poor temporal memory. By showing that memories are moved in time to make room for missing information, the present study demonstrates that narrative time can be adapted to fit a standard template regardless of what has been effectively encoded, in line with reconstructive theories of memory.

Movies and narratives as naturalistic stimuli in neuroimaging

Using movies and narratives as naturalistic stimuli in human neuroimaging studies has yielded significant advances in understanding of cognitive and emotional functions. The relevant literature was reviewed, with emphasis on how the use of naturalistic stimuli has helped advance scientific understanding of human memory, attention, language, emotions, and social cognition in ways that would have been difficult otherwise. These advances include discovering a cortical hierarchy of temporal receptive windows, which supports processing of dynamic information that accumulates over several time scales, such as immediate reactions vs. slowly emerging patterns in social interactions. Naturalistic stimuli have also helped elucidate how the hippocampus supports segmentation and memorization of events in day-to-day life and have afforded insights into attentional brain mechanisms underlying our ability to adopt specific perspectives during natural viewing. Further, neuroimaging studies with naturalistic stimuli have revealed the role of the default-mode network in narrative-processing and in social cognition. Finally, by robustly eliciting genuine emotions, these stimuli have helped elucidate the brain basis of both basic and social emotions apparently manifested as highly overlapping yet distinguishable patterns of brain activity.

Pupil-linked arousal signals track the temporal organization of events in memory

Everyday life unfolds continuously, yet we tend to remember past experiences as discrete event sequences or episodes. Although this phenomenon has been well documented, the neuromechanisms that support the transformation of continuous experience into distinct and memorable episodes remain unknown. Here, we show that changes in context, or event boundaries, elicit a burst of autonomic arousal, as indexed by pupil dilation. Event boundaries also lead to the segmentation of adjacent episodes in later memory, evidenced by changes in memory for the temporal duration, order, and perceptual details of recent event sequences. These subjective and objective changes in temporal memory are also related to distinct temporal features of pupil dilations to boundaries as well as to the temporal stability of more prolonged pupil-linked arousal states. Collectively, our findings suggest that pupil measures reflect both stability and change in ongoing mental context representations, which in turn shape the temporal structure of memory.

Neural and behavioral correlates of episodic memory are associated with temporal discounting in older adults

When facing decisions involving trade-offs between smaller, sooner and larger, delayed rewards, people tend to discount the value of future rewards. There are substantial individual differences in this tendency toward temporal discounting, however. One neurocognitive system that may underlie these individual differences is episodic memory, given the overlap in the neural circuitry involved in imagining the future and remembering the past. Here we tested this hypothesis in older adults, including both those that were cognitively normal and those with amnestic mild cognitive impairment (MCI). We found that performance on neuropsychological measures of episodic memory retrieval was associated with temporal discounting, such that people with better memory discounted delayed rewards less. This relationship was specific to episodic memory and temporal discounting, since executive function (another cognitive ability) was unrelated to temporal discounting, and episodic memory was unrelated to risk tolerance (another decision-making preference). We also examined cortical thickness and volume in medial temporal lobe regions critical for episodic memory. Entorhinal cortical thickness was associated with reduced temporal discounting, with episodic memory performance partially mediating this association. The inclusion of MCI participants was critical to revealing these associations between episodic memory and entorhinal cortical thickness and temporal discounting. These effects were larger in the MCI group, reduced after controlling for MCI status, and statistically significant only when including MCI participants in analyses. Overall, these findings suggest that individual differences in temporal discounting are driven by episodic memory function, and that a decline in medial temporal lobe structural integrity may impact temporal discounting.

Sequence Memory in the Hippocampal-Entorhinal Region

Episodic memories are constructed from sequences of events. When recalling such a memory, we not only recall individual events, but we also retrieve information about how the sequence of events unfolded. Here, we focus on the role of the hippocampal-entorhinal region in processing and remembering sequences of events, which are thought to be stored in relational networks. We summarize evidence that temporal relations are a central organizational principle for memories in the hippocampus. Importantly, we incorporate novel insights from recent studies about the role of the adjacent entorhinal cortex in sequence memory. In rodents, the lateral entorhinal subregion carries temporal information during ongoing behavior. The human homologue is recruited during memory recall where its representations reflect the temporal relationships between events encountered in a sequence. We further introduce the idea that the hippocampal-entorhinal region might enable temporal scaling of sequence representations. Flexible changes of sequence progression speed could underlie the traversal of episodic memories and mental simulations at different paces. In conclusion, we describe how the entorhinal cortex and hippocampus contribute to remembering event sequences-a core component of episodic memory.

Links between autobiographical memory richness and temporal discounting in older adults

When making choices between smaller, sooner rewards and larger, later ones, people tend to discount future outcomes. Individual differences in temporal discounting in older adults have been associated with episodic memory abilities and entorhinal cortical thickness. The cause of this association between better memory and more future-oriented choice remains unclear, however. One possibility is that people with perceptually richer recollections are more patient because they also imagine the future more vividly. Alternatively, perhaps people whose memories focus more on the meaning of events (i.e., are more "gist-based") show reduced temporal discounting, since imagining the future depends on interactions between semantic and episodic memory. We examined which categories of episodic details - perception-based or gist-based - are associated with temporal discounting in older adults. Older adults whose autobiographical memories were richer in perception-based details showed reduced temporal discounting. Furthermore, in an exploratory neuroanatomical analysis, both discount rates and perception-based details correlated with entorhinal cortical thickness. Retrieving autobiographical memories before choice did not affect temporal discounting, however, suggesting that activating episodic memory circuitry at the time of choice is insufficient to alter discounting in older adults. These findings elucidate the role of episodic memory in decision making, which will inform interventions to nudge intertemporal choices.

The hippocampus contributes to temporal duration memory in the context of event sequences: A cross-species perspective

Although a large body of research has implicated the hippocampus in the processing of memory for temporal duration, there is an exigent degree of inconsistency across studies that obfuscates the precise contributions of this structure. To shed light on this issue, the present review article surveys both historical and recent cross-species evidence emanating from a wide variety of experimental paradigms, identifying areas of convergence and divergence. We suggest that while factors such as time-scale (e.g. the length of durations involved) and the nature of memory processing (e.g. prospective vs. retrospective memory) are very helpful in the interpretation of existing data, an additional important consideration is the context in which the duration information is experienced and processed, with the hippocampus being preferentially involved in memory for durations that are embedded within a sequence of events. We consider the mechanisms that may underpin temporal duration memory and how the same mechanisms may contribute to memory for other aspects of event sequences such as temporal order.

Temporal features of spatial knowledge: Representing order and duration of topographical information

Environmental navigation entails the constant integration of information across space and time; however, the relation between spatial and temporal features involved in wayfinding has not been fully established yet. Here we investigated how two key spatio-temporal aspects of navigation - namely the processing of information concerning the order of landmarks along a route, and the duration of tracts connecting the same landmarks - relate to different types of navigational learning. Participants encoded a path in a real city in both a route and a survey format, and the acquisition of landmark, route and survey knowledge was tested. Participants' knowledge of landmarks order, and their perception of tracts duration were also assessed. Performance in the survey task, but not in the landmark and route tasks, significantly predicted accuracy in landmark ordering. The influence of tract length on retrospectively estimated tracts duration was also found to be significantly predicted only by accuracy in the survey learning task. These results support recent models of spatial navigation, invoking the dynamic interaction between different representation formats. Furthermore, they are consistent with theoretical views of an integrated account of the role of the hippocampus in navigation and memory.

Crossing event boundaries changes prospective perceptions of temporal length and proximity

We conducted two experiments to investigate how crossing a single naturalistic event boundary impacted two different types of temporal estimation involving the same target duration - one where participants directly compared marked temporal durations and another where they judged the temporal proximity of stimuli. In Experiment 1, participants judged whether time intervals presented during movies of everyday events were shorter or longer than a previously encoded 5-s reference interval. We examined how the presence of a transition between events (event boundary) in the movie influenced people's judgments about the length of the comparison interval. Comparison intervals presented during a portion of the movie containing an event boundary were judged as shorter than the reference interval more often than comparison intervals that contained no boundary. Working-memory updating at the event boundary may have directed attention away from the concurrent timing task. In Experiment 2, participants judged whether the second of three tones presented during everyday movies was closer to the first or the third tone presented. Tones separated by an event boundary were judged as farther apart than tones contained within the same event. When judging temporal proximity, attention directed to processing information at an event boundary between two stimuli may disrupt the formation of temporal associations between those stimuli. Overall, these results demonstrate that crossing a single event boundary can impact people's prospective perceptions of the temporal characteristics of their experience and suggest that the episodic memory updating that occurs during an event boundary both captures timing-relevant attentional resources and plays a role in the temporal binding of information.

Mapping sequence structure in the human lateral entorhinal cortex

Remembering event sequences is central to episodic memory and presumably supported by the hippocampal-entorhinal region. We previously demonstrated that the hippocampus maps spatial and temporal distances between events encountered along a route through a virtual city (Deuker et al., 2016), but the content of entorhinal mnemonic representations remains unclear. Here, we demonstrate that multi-voxel representations in the anterior-lateral entorhinal cortex (alEC) - the human homologue of the rodent lateral entorhinal cortex - specifically reflect the temporal event structure after learning. Holistic representations of the sequence structure related to memory recall and the timeline of events could be reconstructed from entorhinal multi-voxel patterns. Our findings demonstrate representations of temporal structure in the alEC; dovetailing with temporal information carried by population signals in the lateral entorhinal cortex of navigating rodents and alEC activations during temporal memory retrieval. Our results provide novel evidence for the role of the alEC in representing time for episodic memory.

Episodic memory: Neuronal codes for what, where, and when

Episodic memory is defined as the ability to recall events in a spatiotemporal context. Formation of such memories is critically dependent on the hippocampal formation and its inputs from the entorhinal cortex. To be able to support the formation of episodic memories, entorhinal cortex and hippocampal formation should contain a neuronal code that follows several requirements. First, the code should include information about position of the agent ("where"), sequence of events ("when"), and the content of the experience itself ("what"). Second, the code should arise instantly thereby being able to support memory formation of one-shot experiences. For successful encoding and to avoid interference between memories during recall, variations in location, time, or in content of experience should result in unique ensemble activity. Finally, the code should capture several different resolutions of experience so that the necessary details relevant for future memory-based predictions will be stored. We review how neuronal codes in entorhinal cortex and hippocampus follow these requirements and argue that during formation of episodic memories entorhinal cortex provides hippocampus with instant information about ongoing experience. Such information originates from (a) spatially modulated neurons in medial entorhinal cortex, including grid cells, which provide a stable and universal positional metric of the environment; (b) a continuously varying signal in lateral entorhinal cortex providing a code for the temporal progression of events; and (c) entorhinal neurons coding the content of experiences exemplified by object-coding and odor-selective neurons. During formation of episodic memories, information from these systems are thought to be encoded as unique sequential ensemble activity in hippocampus, thereby encoding associations between the content of an event and its spatial and temporal contexts. Upon exposure to parts of the encoded stimuli, activity in these ensembles can be reinstated, leading to reactivation of the encoded activity pattern and memory recollection.

Entorhinal transformations in abstract frames of reference

Knowing how another's preferences relate to our own is a central aspect of everyday decision-making, yet how the brain performs this transformation is unclear. Here, we ask whether the putative role of the hippocampal-entorhinal system in transforming relative and absolute spatial coordinates during navigation extends to transformations in abstract decision spaces. During functional magnetic resonance imaging (fMRI), subjects learned a stranger's preference for an everyday activity-relative to one of three personally known individuals-and subsequently decided how the stranger's preference relates to the other two individuals' preferences. We observed entorhinal/subicular responses to the absolute distance between the ratings of the stranger and the familiar choice options. Notably, entorhinal/subicular signals were sensitive to which familiar individuals were being compared to the stranger. In contrast, striatal signals increased when accurately determining the ordinal position of choice options in relation to the stranger. Paralleling its role in navigation, these data implicate the entorhinal/subicular region in assimilating relatively coded knowledge within abstract metric spaces.

Transcending time in the brain: How event memories are constructed from experience

Our daily lives unfold continuously, yet when we reflect on the past, we remember those experiences as distinct and cohesive events. To understand this phenomenon, early investigations focused on how and when individuals perceive natural breakpoints, or boundaries, in ongoing experience. More recent research has examined how these boundaries modulate brain mechanisms that support long-term episodic memory. This work has revealed that a complex interplay between hippocampus and prefrontal cortex promotes the integration and separation of sequential information to help organize our experiences into mnemonic events. Here, we discuss how both temporal stability and change in one's thoughts, goals, and surroundings may provide scaffolding for these neural processes to link and separate memories across time. When learning novel or familiar sequences of information, dynamic hippocampal processes may work both independently from and in concert with other brain regions to bind sequential representations together in memory. The formation and storage of discrete episodic memories may occur both proactively as an experience unfolds. They may also occur retroactively, either during a context shift or when reactivation mechanisms bring the past into the present to allow integration. We also describe conditions and factors that shape the construction and integration of event memories across different timescales. Together these findings shed new light on how the brain transcends time to transform everyday experiences into meaningful memory representations.

Precise temporal memories are supported by the lateral entorhinal cortex in humans

There is accumulating evidence that the entorhinal-hippocampal network is important for temporal memory. However, relatively little is known about the precise neurobiological mechanisms underlying memory for time. In particular, whether the lateral entorhinal cortex (LEC) is involved in temporal processing remains an open question. During high-resolution functional magnetic resonance imaging (fMRI) scanning, participants watched a ~28-min episode of a television show. During the test, they viewed still-frames and indicated on a continuous timeline the precise time each still-frame was viewed during the study. This procedure allowed us to measure error in seconds for each trial. We analyzed fMRI data from retrieval and found that high temporal precision was associated with increased blood-oxygen-level-dependent fMRI activity in the anterolateral entorhinal (a homolog of the LEC in rodents) and perirhinal cortices, but not in the posteromedial entorhinal and parahippocampal cortices. This suggests a previously unknown role for the LEC in processing of high-precision, minute-scale temporal memories.

Retrospective time estimation following damage to the prefrontal cortex

Time estimation in patients with prefrontal cortex (PFC) damage is often inaccurate. The relationship between PFC and estimation of short time intervals has been examined. However, it remains unclear whether PFC damage affects estimation of longer time intervals. Here, we investigated the ability of patients and healthy subjects to verbally estimate a period of 30 min, using a method easily applied in clinical settings. In 99 patients with brain damage, we compared under and normal ranges of time in patients with PFC damage or damage to other brain areas with the chi-squared test. Subsequently, we conducted a discriminant analysis and a multiple linear regression analysis to identify specific brain areas affecting time estimation. We observed a significantly larger number of patients who overestimated 30 min in the group with bilateral PFC damage compared to patients with damage to other regions. Discriminant analysis revealed that damage of right lateral PFC and left medial PFC contributed to discrimination between the normal range and overestimation groups. Multiple linear regression analysis indicated that right lateral PFC damage strongly affected overestimation of a 30-min interval. Neuropsychological test results revealed lower general cognitive function scores and orientation scores in overestimation group. The length of estimated time and the score of delayed word recall were negatively correlated. We propose that these may require encoding, maintenance, and updating of memory and are indirectly related to contextual memory. We discuss hypotheses on contextual memory segmentation and reconstruction to clarify the mechanism of impaired time overestimation in PFC-damaged patients.

Multivoxel pattern similarity suggests the integration of temporal duration in hippocampal event sequence representations

Recent rodent work suggests the hippocampus may provide a temporal representation of event sequences, in which the order of events and the interval durations between them are encoded. There is, however, limited human evidence for the latter, in particular whether the hippocampus processes duration information pertaining to the passage of time rather than qualitative or quantitative changes in event content. We scanned participants while they made match-mismatch judgements on each trial between a study sequence of events and a subsequent test sequence. Participants explicitly remembered event order or interval duration information (Experiment 1), or monitored order only, with duration being manipulated implicitly (Experiment 2). Hippocampal study-test pattern similarity was significantly reduced by changes to order or duration in mismatch trials, even when duration was processed implicitly. Our findings suggest the human hippocampus processes short intervals within sequences and support the idea that duration information is integrated into hippocampal mnemonic representations.