Cortical and subcortical contributions to sequence retrieval: Schematic coding of temporal context in the neocortical recollection network

The history of episodic memory

Over the course of his research, Endel Tulving offered a number of somewhat different characterizations of episodic memory. Do they indicate that he changed his mind over time as to what episodic memory is, or did his core understanding of the nature of episodic memory stay the same? In this article, we offer some support for the latter claim, and in particular for thinking that, throughout his life, Tulving took as a defining feature of episodic memory the distinctive awareness of the self in time it involves. We argue that it is easier to see the continuities rather than the discontinuities in Tulving's writings once their historical context is taken into account, where this involves both the authors who influenced his thinking, as well as the intellectual climate at the different times he was writing. We also discuss two recent bodies of work on episodic memory that take aspects of Tulving's writings as their point of departure, but try to factor out into separate ingredients what he arguably saw as a unitary phenomenon. Considering aspects of the dialectic between them and Tulving's view might shed further light on some of the motivations behind the latter.This article is part of the theme issue 'Elements of episodic memory: lessons from 40 years of research'.

Consolidation Enhances Sequential Multistep Anticipation but Diminishes Access to Perceptual Features

Many experiences unfold predictably over time. Memory for these temporal regularities enables anticipation of events multiple steps into the future. Because temporally predictable events repeat over days, weeks, and years, we must maintain-and potentially transform-memories of temporal structure to support adaptive behavior. We explored how individuals build durable models of temporal regularities to guide multistep anticipation. Healthy young adults (Experiment 1: N = 99, age range = 18-40 years; Experiment 2: N = 204, age range = 19-40 years) learned sequences of scene images that were predictable at the category level and contained incidental perceptual details. Individuals then anticipated upcoming scene categories multiple steps into the future, immediately and at a delay. Consolidation increased the efficiency of anticipation, particularly for events further in the future, but diminished access to perceptual features. Further, maintaining a link-based model of the sequence after consolidation improved anticipation accuracy. Consolidation may therefore promote efficient and durable models of temporal structure, thus facilitating anticipation of future events.

Episodic boundaries affect neural features of representational drift in humans

A core feature of episodic memory is representational drift, the gradual change in aggregate oscillatory features that supports temporal association of memory items. However, models of drift overlook the role of episodic boundaries, which indicate a shift from prior to current context states. Our study focuses on the impact of task boundaries on representational drift in the parietal and temporal lobes in 99 subjects during a free recall task. Using intracranial EEG recordings, we show boundary representations reset gamma band drift in the medial parietal lobe, selectively enhancing the recall of early list (primacy) items. Conversely, the lateral temporal cortex shows increased drift for recalled items but lacked sensitivity to task boundaries. Our results suggest regional sensitivity to varied contextual features: the lateral temporal cortex uses drift to differentiate items, while the medial parietal lobe uses drift-resets to associate items with the current context. We propose drift represents relational information tailored to a region's sensitivity to unique contextual elements. Our findings offer a mechanism to integrate models of temporal association by drift with event segmentation by episodic boundaries.

Investigating the impact of healthy aging on memory for temporal duration and order

Temporal information, including information about temporal order and duration, is a fundamental component of event sequence memory. While previous research has demonstrated that aging can have a detrimental effect on memory for temporal order, there has been limited insight into the effect of aging on memory for durations, particularly within the context of sequences. In the current study, neurologically healthy young and older participants were administered two temporal match-mismatch tasks: one in which they were instructed on each trial to compare the temporal order or duration information of stimulus sequences presented first in a study phase and then, after a short delay, in a test phase (event sequence task); and a second in which participants were required to compare single durations or sequences of durations across study and test phases of each trial (pinwheel task). Consistent with the literature, the older participants were significantly poorer compared to their younger counterparts at making temporal order match-mismatch judgments in the event sequence task. In addition to this, data from both tasks suggested that the older adults were also less accurate at match-mismatch judgments based on duration information, with tentative evidence from the pinwheel task to suggest that this age-related effect was most prominent when the duration information was presented within a sequence. We suggest that age-related changes to medial temporal and frontal lobe function may contribute to changes in memory for temporal information in older adults, given the importance of these regions to event sequence memory.

Sequential order spatial memory in male rats: Characteristics and impact of medial prefrontal cortex and hippocampus disruption

Remembering an experience entails linking what happened, where the event transpired, and when it occurred. Most rodent hippocampal studies involve tests of spatial memory, but fewer investigate temporal and sequential order memory. Here we provide a demonstration of rats learning an aversive sequential order task using a radial arm water maze. Male rats learned a fixed sequence of up to seven spatial locations, with each decision session separated by a temporal delay. Rats relied on visuospatial cues and the number of times they had entered the maze for a given day in order to successfully perform the task. Behavioral patterns during asymptotic performance showed similarities to the serial-position effect, especially with regards to faster first choice latency. Rats at asymptotic performance were implanted with bilateral cannula in medial prefrontal cortex, dorsal, and ventral hippocampus. After re-training, we injected muscimol to temporarily disrupt targeted brain regions. While control rats made prospective errors, rats with mPFC muscimol exhibited more retrospective errors. Rats with hippocampal muscimol no longer exhibited a prospective bias and were at chance levels in their error choices. Taken together, our results suggest disruption of mPFC, but not the hippocampus, produced an error choice bias during an aversive sequential order spatial processing task.

Rethinking retrosplenial cortex: Perspectives and predictions

The last decade has produced exciting new ideas about retrosplenial cortex (RSC) and its role in integrating diverse inputs. Here, we review the diversity in forms of spatial and directional tuning of RSC activity, temporal organization of RSC activity, and features of RSC interconnectivity with other brain structures. We find that RSC anatomy and dynamics are more consistent with roles in multiple sensorimotor and cognitive processes than with any isolated function. However, two more generalized categories of function may best characterize roles for RSC in complex cognitive processes: (1) shifting and relating perspectives for spatial cognition and (2) prediction and error correction for current sensory states with internal representations of the environment. Both functions likely take advantage of RSC's capacity to encode conjunctions among sensory, motor, and spatial mapping information streams. Together, these functions provide the scaffold for intelligent actions, such as navigation, perspective taking, interaction with others, and error detection.

Memory-Based Prediction Deficits and Dorsolateral Prefrontal Dysfunction in Schizophrenia

BACKGROUND

Theories suggest that people with schizophrenia (SZ) have problems generating predictions based on past experiences. The dorsolateral prefrontal cortex (DLPFC) and hippocampus participate in memory-based prediction. We used functional magnetic resonance imaging to investigate DLPFC and hippocampal function in healthy control (HC) subjects and people with SZ during memory-based prediction.

METHODS

Prior to scanning, HC subjects (n = 54) and people with SZ (n = 31) learned 5-object sequences presented in fixed or random orders on each repetition. During scanning, participants made semantic decisions (e.g., "Can this object fit in a shoebox?") on a continuous stream of objects from fixed and random sequences. Sequence prediction was demonstrated by faster semantic decisions for objects in fixed versus random sequences because memory could be used to anticipate and more efficiently process semantic information about upcoming objects in fixed sequences. Representational similarity analyses were used to determine how each sequence type was represented in the posterior hippocampus and DLPFC.

RESULTS

Sequence predictions were reduced in individuals with SZ relative to HC subjects. Representational similarity analyses revealed stronger memory-based predictions in the DLPFC of HC subjects than people with SZ, and DLPFC representations correlated with more successful predictions in HC subjects only. For the posterior hippocampus, voxel pattern similarity was increased for fixed versus random sequences in HC subjects only, but no significant between-group differences or correlations with prediction success were observed.

CONCLUSIONS

Individuals with SZ are capable of learning temporal sequences; however, they are impaired using memory to predict upcoming events as efficiently as HC subjects. This deficit appears related to disrupted neural representation of sequence information in the DLPFC.

The neural bases for timing of durations

Durations are defined by a beginning and an end, and a major distinction is drawn between durations that start in the present and end in the future ('prospective timing') and durations that start in the past and end either in the past or the present ('retrospective timing'). Different psychological processes are thought to be engaged in each of these cases. The former is thought to engage a clock-like mechanism that accurately tracks the continuing passage of time, whereas the latter is thought to engage a reconstructive process that utilizes both temporal and non-temporal information from the memory of past events. We propose that, from a biological perspective, these two forms of duration 'estimation' are supported by computational processes that are both reliant on population state dynamics but are nevertheless distinct. Prospective timing is effectively carried out in a single step where the ongoing dynamics of population activity directly serve as the computation of duration, whereas retrospective timing is carried out in two steps: the initial generation of population state dynamics through the process of event segmentation and the subsequent computation of duration utilizing the memory of those dynamics.

Assessing mild cognitive impairment using object-location memory in immersive virtual environments

Pathological changes in the medial temporal lobe (MTL) are found in the early stages of Alzheimer's disease (AD) and aging. The earliest pathological accumulation of tau colocalizes with the areas of the MTL involved in object processing as part of a wider anterolateral network. Here, we sought to assess the diagnostic potential of memory for object locations in iVR environments in individuals at high risk of AD dementia (amnestic mild cognitive impairment [aMCI] n = 23) as compared to age-related cognitive decline. Consistent with our primary hypothesis that early AD would be associated with impaired object location, aMCI patients exhibited impaired spatial feature binding. Compared to both older (n = 24) and younger (n = 53) controls, aMCI patients, recalled object locations with significantly less accuracy (p < .001), with a trend toward an impaired identification of the object's correct context (p = .05). Importantly, these findings were not explained by deficits in object recognition (p = .6). These deficits differentiated aMCI from controls with greater accuracy (AUC = 0.89) than the standard neuropsychological tests. Within the aMCI group, 16 had CSF biomarkers indicative of their likely AD status (MCI+ n = 9 vs. MCI- n = 7). MCI+ showed lower accuracy in the object-context association than MCI- (p = .03) suggesting a selective deficit in object-context binding postulated to be associated with anterior-temporal areas. MRI volumetric analysis across healthy older participants and aMCI revealed that test performance positively correlates with lateral entorhinal cortex volumes (p < .05) and hippocampus volumes (p < .01), consistent with their hypothesized role in binding contextual and spatial information with object identity. Our results indicate that tests relying on the anterolateral object processing stream, and in particular requiring successful binding of an object with spatial information, may aid detection of pre-dementia AD due to the underlying early spread of tau pathology.

Mnemonic construction and representation of temporal structure in the hippocampal formation

The hippocampal-entorhinal region supports memory for episodic details, such as temporal relations of sequential events, and mnemonic constructions combining experiences for inferential reasoning. However, it is unclear whether hippocampal event memories reflect temporal relations derived from mnemonic constructions, event order, or elapsing time, and whether these sequence representations generalize temporal relations across similar sequences. Here, participants mnemonically constructed times of events from multiple sequences using infrequent cues and their experience of passing time. After learning, event representations in the anterior hippocampus reflected temporal relations based on constructed times. Temporal relations were generalized across sequences, revealing distinct representational formats for events from the same or different sequences. Structural knowledge about time patterns, abstracted from different sequences, biased the construction of specific event times. These findings demonstrate that mnemonic construction and the generalization of relational knowledge combine in the hippocampus, consistent with the simulation of scenarios from episodic details and structural knowledge.

Distinct cortical systems reinstate the content and context of episodic memories

Episodic recall depends upon the reinstatement of cortical activity present during the formation of a memory. Evidence from functional neuroimaging and invasive recordings in humans suggest that reinstatement organizes our memories by time or content, yet the neural systems involved in reinstating these unique types of information remain unclear. Here, combining computational modeling and intracranial recordings from 69 epilepsy patients, we show that two cortical systems uniquely reinstate the semantic content and temporal context of previously studied items during free recall. Examining either the posterior medial or anterior temporal networks, we find that forward encoding models trained on the brain's response to the temporal and semantic attributes of items can predict the serial position and semantic category of unseen items. During memory recall, these models uniquely link reinstatement of temporal context and semantic content to these posterior and anterior networks, respectively. These findings demonstrate how specialized cortical systems enable the human brain to target specific memories.

Activation of Person Knowledge in Medial Prefrontal Cortex during the Encoding of New Lifelike Events

Our knowledge about people can help us predict how they will behave in particular situations and interpret their actions. In this study, we investigated the cognitive and neural effects of person knowledge on the encoding and retrieval of novel life-like events. Healthy human participants learnt about two characters over a week by watching 6 episodes of one of two situation comedies, which were both centered on a young couple. In the scanner, they watched and then silently recalled 20 new scenes from both shows that were all set in unfamiliar locations: 10 from their trained show and 10 from the untrained show. After scanning, participants' recognition memory was better for scenes from the trained show. The functional magnetic resonance imaging (fMRI) patterns of brain activity when watching the videos were reinstated during recall, but this effect was not modulated by training. However, person knowledge boosted the similarity in fMRI patterns of activity in the medial prefrontal cortex (MPFC) when watching the new events involving familiar characters. Our findings identify a role for the MPFC in the representation of schematic person knowledge during the encoding of novel, lifelike events.

Imaging recollection, familiarity, and novelty in the frontoparietal control and default mode networks and the anterior-posterior medial temporal lobe: An integrated view and meta-analysis

A network-level model of recollection-based recognition (R), familiarity-based recognition (F), and novelty recognition (N) was constructed, and its validity was evaluated through meta-analyses to produce an integrated view of neuroimaging data. The model predicted the following: (a) the overall magnitude of the frontoparietal control network (FPCN) activity (which supports retrieval and decision effort) is in the order of F > R > N; (b) that of the posterior medial temporal network (MTL) activity (which plays a direct role in retrieval) is in the order of R > N > F; (c) that of the anterior MTL activity (which supports novelty-encoding) is in the order of N > R > F; (d) that of the default mode network (DMN) activity (which supports the subjective experience of remembering) is in the order of R > N > F. The meta-analyses results were consistent with these predictions. Subsystem analysis indicated a functional dissociation between the cingulo-opercular vs. frontoparietal components of the FPCN and between the core vs. medial temporal components of the DMN.

Medial prefrontal cortex and hippocampal activity differentially contribute to ordinal and temporal context retrieval during sequence memory

Remembering sequences of events defines episodic memory, but retrieval can be driven by both ordinality and temporal contexts. Whether these modes of retrieval operate at the same time or not remains unclear. Theoretically, medial prefrontal cortex (mPFC) confers ordinality, while the hippocampus (HC) associates events in gradually changing temporal contexts. Here, we looked for evidence of each with BOLD fMRI in a sequence task that taxes both retrieval modes. To test ordinal modes, items were transferred between sequences but retained their position (e.g., AB3). Ordinal modes activated mPFC, but not HC. To test temporal contexts, we examined items that skipped ahead across lag distances (e.g., ABD). HC, but not mPFC, tracked temporal contexts. There was a mPFC and HC by retrieval mode interaction. These current results suggest that the mPFC and HC are concurrently engaged in different retrieval modes in support of remembering when an event occurred.

Hierarchical Representation of Multistep Tasks in Multiple-Demand and Default Mode Networks

Task episodes consist of sequences of steps that are performed to achieve a goal. We used fMRI to examine neural representation of task identity, component items, and sequential position, focusing on two major cortical systems—the multiple-demand (MD) and default mode networks (DMN). Human participants (20 males, 22 females) learned six tasks each consisting of four steps. Inside the scanner, participants were cued which task to perform and then sequentially identified the target item of each step in the correct order. Univariate time course analyses indicated that intra-episode progress was tracked by a tonically increasing global response, plus an increasing phasic step response specific to MD regions. Inter-episode boundaries evoked a widespread response at episode onset, plus a marked offset response specific to DMN regions. Representational similarity analysis (RSA) was used to examine representation of task identity and component steps. Both networks represented the content and position of individual steps, however the DMN preferentially represented task identity while the MD network preferentially represented step-level information. Thus, although both MD and DMN networks are sensitive to step-level and episode-level information in the context of hierarchical task performance, they exhibit dissociable profiles in terms of both temporal dynamics and representational content. The results suggest collaboration of multiple brain regions in control of multistep behavior, with MD regions particularly involved in processing the detail of individual steps, and DMN adding representation of broad task context.

SIGNIFICANCE STATEMENT Achieving one's goals requires knowing what to do and when. Tasks are typically hierarchical, with smaller steps nested within overarching goals. For effective, flexible behavior, the brain must represent both levels. We contrast response time courses and information content of two major cortical systems—the multiple-demand (MD) and default mode networks (DMN)—during multistep task episodes. Both networks are sensitive to step-level and episode-level information, but with dissociable profiles. Intra-episode progress is tracked by tonically increasing global responses, plus MD-specific increasing phasic step responses. Inter-episode boundaries evoke widespread responses at episode onset, plus DMN-specific offset responses. Both networks represent content and position of individual steps; however, the DMN and MD networks favor task identity and step-level information, respectively.

Sequence Memory in the Hippocampal-Entorhinal Region

Episodic memories are constructed from sequences of events. When recalling such a memory, we not only recall individual events, but we also retrieve information about how the sequence of events unfolded. Here, we focus on the role of the hippocampal-entorhinal region in processing and remembering sequences of events, which are thought to be stored in relational networks. We summarize evidence that temporal relations are a central organizational principle for memories in the hippocampus. Importantly, we incorporate novel insights from recent studies about the role of the adjacent entorhinal cortex in sequence memory. In rodents, the lateral entorhinal subregion carries temporal information during ongoing behavior. The human homologue is recruited during memory recall where its representations reflect the temporal relationships between events encountered in a sequence. We further introduce the idea that the hippocampal-entorhinal region might enable temporal scaling of sequence representations. Flexible changes of sequence progression speed could underlie the traversal of episodic memories and mental simulations at different paces. In conclusion, we describe how the entorhinal cortex and hippocampus contribute to remembering event sequences-a core component of episodic memory.

The hippocampus, prefrontal cortex, and perirhinal cortex are critical to incidental order memory

Considerable research in rodents and humans indicates the hippocampus and prefrontal cortex are essential for remembering temporal relationships among stimuli, and accumulating evidence suggests the perirhinal cortex may also be involved. However, experimental parameters differ substantially across studies, which limits our ability to fully understand the fundamental contributions of these structures. In fact, previous studies vary in the type of temporal memory they emphasize (e.g., order, sequence, or separation in time), the stimuli and responses they use (e.g., trial-unique or repeated sequences, and incidental or rewarded behavior), and the degree to which they control for potential confounding factors (e.g., primary and recency effects, or order memory deficits secondary to item memory impairments). To help integrate these findings, we developed a new paradigm testing incidental memory for trial-unique series of events, and concurrently assessed order and item memory in animals with damage to the hippocampus, prefrontal cortex, or perirhinal cortex. We found that this new approach led to robust order and item memory, and that hippocampal, prefrontal and perirhinal damage selectively impaired order memory. These findings suggest the hippocampus, prefrontal cortex and perirhinal cortex are part of a broad network of structures essential for incidentally learning the order of events in episodic memory.

The contributions of spatial context and imagery to the recollection of single words

A number of theories of hippocampal function have placed spatial context at the center of richly recollected memories, but the subjective and objective ways that spatial context underlies the recollection of single words has been largely overlooked and underexplained. In this study, we conducted three experiments to investigate the involvement of spatial context in the recollection of single words. In all three experiments, participants encoded single words with varying features such as location and color. The subjective experience of recollection was measured using remember/know judgments and participant self-report of the types of information they recollected about the words. Objectively, recollection was measured using source memory judgments for both spatial and non-spatial features associated with the words. Our results provide evidence that spatial context frequently accompanies the recollection of single, isolated words, reviving discussions on the role of the hippocampus in spatial and detailed recollection.

Prefrontal Pathways Provide Top-Down Control of Memory for Sequences of Events

We remember our lives as sequences of events, but it is unclear how these memories are controlled during retrieval. In rats, the medial prefrontal cortex (mPFC) is positioned to influence sequence memory through extensive top-down inputs to regions heavily interconnected with the hippocampus, notably the nucleus reuniens of the thalamus (RE) and perirhinal cortex (PER). Here, we used an hM4Di synaptic-silencing approach to test our hypothesis that specific mPFC→RE and mPFC→PER projections regulate sequence memory retrieval. First, we found non-overlapping populations of mPFC cells project to RE and PER. Second, suppressing mPFC activity impaired sequence memory. Third, inhibiting mPFC→RE and mPFC→PER pathways effectively abolished sequence memory. Finally, a sequential lag analysis showed that the mPFC→RE pathway contributes to a working memory retrieval strategy, whereas the mPFC→PER pathway supports a temporal context memory retrieval strategy. These findings demonstrate that mPFC→RE and mPFC→PER pathways serve as top-down mechanisms that control distinct sequence memory retrieval strategies.

Tagged MEG measures binocular rivalry in a cortical network that predicts alternation rate

Binocular rivalry (BR) is a dynamic visual illusion that provides insight into the cortical mechanisms of visual awareness, stimulus selection, and object identification. When dissimilar binocular images cannot be fused, perception switches every few seconds between the left and right eye images. The speed at which individuals switch between alternatives is a stable, partially heritable trait. In order to isolate the monocular and binocular processes that determine the speed of rivalry, we presented stimuli tagged with a different flicker frequency in each eye and applied stimulus-phase locked MEG source imaging. We hypothesized that the strength of the evoked fundamental or intermodulation frequencies would vary when comparing Fast and Slow Switchers. Ten subjects reported perceptual alternations, with mean dominance durations between 1.2–4.0 sec. During BR, event-related monocular input in V1, and broadly in higher-tier ventral temporal cortex, waxed and waned with the periods of left or right eye dominance/suppression. In addition, we show that Slow Switchers produce greater evoked intermodulation frequency responses in a cortical network composed of V1, lateral occipital, posterior STS, retrosplenial & superior parietal cortices. Importantly, these dominance durations were not predictable from the brain responses to either of the fundamental tagging frequencies in isolation, nor from any responses to a pattern rivalry control condition, or a non-rivalrous control. The novel cortical network isolated, which overlaps with the default-mode network, may contain neurons that compute the level of endogenous monocular difference, and monitor accumulation of this conflict over extended periods of time. These findings are the first to relate the speed of rivalry across observers to the ‘efficient coding’ theory of computing binocular differences that may apply to binocular vision generally.

Rhythmic Temporal Structure at Encoding Enhances Recognition Memory

Presenting events in a rhythm has been shown to enhance perception and facilitate responses for stimuli that appear in synchrony with the rhythm, but little is known about how rhythm during encoding influences later recognition. In this study, participants were presented with images of everyday objects in an encoding phase before a recognition task in which they judged whether or not objects were previously presented. Blockwise, object presentation during encoding followed either a rhythmic (constant, predictable) or arrhythmic (random, unpredictable) temporal structure, of which participants were unaware. Recognition was greater for items presented in a rhythmic relative to an arrhythmic manner. During encoding, there was a differential neural activity based on memory effect with larger positivity for rhythmic over arrhythmic stimuli. At recognition, memory-specific ERP components were differentially affected by temporal structure: The FN400 old/new effect was unaffected by rhythmic structure, whereas the late positive component old/new effect was observed only for rhythmically encoded items. Taken together, this study provides new evidence that memory-specific processing at recognition is affected by temporal structure at encoding.

Medial Orbitofrontal Cortex, Dorsolateral Prefrontal Cortex, and Hippocampus Differentially Represent the Event Saliency

Two primary functions attributed to the hippocampus and prefrontal cortex (PFC) network are retaining the temporal and spatial associations of events and detecting deviant events. It is unclear, however, how these two functions converge into one mechanism. Here, we tested whether increased activity with perceiving salient events is a deviant detection signal or contains information about the event associations by reflecting the magnitude of deviance (i.e., event saliency). We also tested how the deviant detection signal is affected by the degree of anticipation. We studied regional neural activity when people watched a movie that had varying saliency of a novel or an anticipated flow of salient events. Using intracranial electroencephalography from 10 patients, we observed that high-frequency activity (50-150 Hz) in the hippocampus, dorsolateral PFC, and medial OFC tracked event saliency. We also observed that medial OFC activity was stronger when the salient events were anticipated than when they were novel. These results suggest that dorsolateral PFC and medial OFC, as well as the hippocampus, signify the saliency magnitude of events, reflecting the hierarchical structure of event associations.

Transcending time in the brain: How event memories are constructed from experience

Our daily lives unfold continuously, yet when we reflect on the past, we remember those experiences as distinct and cohesive events. To understand this phenomenon, early investigations focused on how and when individuals perceive natural breakpoints, or boundaries, in ongoing experience. More recent research has examined how these boundaries modulate brain mechanisms that support long-term episodic memory. This work has revealed that a complex interplay between hippocampus and prefrontal cortex promotes the integration and separation of sequential information to help organize our experiences into mnemonic events. Here, we discuss how both temporal stability and change in one's thoughts, goals, and surroundings may provide scaffolding for these neural processes to link and separate memories across time. When learning novel or familiar sequences of information, dynamic hippocampal processes may work both independently from and in concert with other brain regions to bind sequential representations together in memory. The formation and storage of discrete episodic memories may occur both proactively as an experience unfolds. They may also occur retroactively, either during a context shift or when reactivation mechanisms bring the past into the present to allow integration. We also describe conditions and factors that shape the construction and integration of event memories across different timescales. Together these findings shed new light on how the brain transcends time to transform everyday experiences into meaningful memory representations.

Sequential Control Underlies Robust Ramping Dynamics in the Rostrolateral Prefrontal Cortex

An essential human skill is our capacity to monitor and execute a sequence of tasks in the service of an overarching goal. Such a sequence can be as mundane as making a cup of coffee or as complex as flying a fighter plane. Previously, we showed that, during sequential control, the rostrolateral prefrontal cortex (RLPFC) exhibits activation that ramps steadily through the sequence and is necessary for sequential task execution using fMRI in humans (Desrochers et al., 2015). It remains unknown what computations may underlie this ramping dynamic. Across two independent fMRI experiments, we manipulated three features that were unique to the sequential control task to determine whether and how they modulated ramping activity in the RLPFC: (1) sequence position uncertainty, (2) sequential monitoring without external position cues (i.e., from memory), and (3) sequential monitoring without multilevel decision making (i.e., task execution). We replicated the ramping activation in RLPFC and found it to be remarkably robust regardless of the level of task abstraction or engagement of memory functions. Therefore, these results both replicate and extend previous findings regarding the function of the RLPFC. They suggest that sequential control processes are integral to the dynamics of RLPFC activity. Advancing knowledge of the neural bases of sequential control is crucial for our understanding of the sequential processes that are necessary for daily living.SIGNIFICANCE STATEMENT We perform sequences of tasks every day, but little is known about how they are controlled in the brain. Previously we found that ramping activity in the rostrolateral prefrontal cortex (RLPFC) was necessary to perform a sequence of tasks. We designed two independent fMRI experiments in human participants to determine which features of the previous sequential task potentially engaged ramping in the RLPFC. We found that any demand to monitor a sequence of state transitions consistently elicited ramping in the RLPFC, regardless of the level of the decisions made at each step in the sequence or engagement of memory functions. These results provide a framework for understanding RLPFC function during sequential control, and consequently, daily life.

Verbal Paired Associates and the Hippocampus: The Role of Scenes

It is widely agreed that patients with bilateral hippocampal damage are impaired at binding pairs of words together. Consequently, the verbal paired associates (VPA) task has become emblematic of hippocampal function. This VPA deficit is not well understood and is particularly difficult for hippocampal theories with a visuospatial bias to explain (e.g., cognitive map and scene construction theories). Resolving the tension among hippocampal theories concerning the VPA could be important for leveraging a fuller understanding of hippocampal function. Notably, VPA tasks typically use high imagery concrete words and so conflate imagery and binding. To determine why VPA engages the hippocampus, we devised an fMRI encoding task involving closely matched pairs of scene words, pairs of object words, and pairs of very low imagery abstract words. We found that the anterior hippocampus was engaged during processing of both scene and object word pairs in comparison to abstract word pairs, despite binding occurring in all conditions. This was also the case when just subsequently remembered stimuli were considered. Moreover, for object word pairs, fMRI activity patterns in anterior hippocampus were more similar to those for scene imagery than object imagery. This was especially evident in participants who were high imagery users and not in mid and low imagery users. Overall, our results show that hippocampal engagement during VPA, even when object word pairs are involved, seems to be evoked by scene imagery rather than binding. This may help to resolve the issue that visuospatial hippocampal theories have in accounting for verbal memory.

Representation of Real-World Event Schemas during Narrative Perception

Understanding movies and stories requires maintaining a high-level situation model that abstracts away from perceptual details to describe the location, characters, actions, and causal relationships of the currently unfolding event. These models are built not only from information present in the current narrative, but also from prior knowledge about schematic event scripts, which describe typical event sequences encountered throughout a lifetime. We analyzed fMRI data from 44 human subjects (male and female) presented with 16 three-minute stories, consisting of four schematic events drawn from two different scripts (eating at a restaurant or going through the airport). Aside from this shared script structure, the stories varied widely in terms of their characters and storylines, and were presented in two highly dissimilar formats (audiovisual clips or spoken narration). One group was presented with the stories in an intact temporal sequence, while a separate control group was presented with the same events in scrambled order. Regions including the posterior medial cortex, medial prefrontal cortex (mPFC), and superior frontal gyrus exhibited schematic event patterns that generalized across stories, subjects, and modalities. Patterns in mPFC were also sensitive to overall script structure, with temporally scrambled events evoking weaker schematic representations. Using a Hidden Markov Model, patterns in these regions predicted the script (restaurant vs airport) of unlabeled data with high accuracy and were used to temporally align multiple stories with a shared script. These results extend work on the perception of controlled, artificial schemas in human and animal experiments to naturalistic perception of complex narratives.SIGNIFICANCE STATEMENT In almost all situations we encounter in our daily lives, we are able to draw on our schematic knowledge about what typically happens in the world to better perceive and mentally represent our ongoing experiences. In contrast to previous studies that investigated schematic cognition using simple, artificial associations, we measured brain activity from subjects watching movies and listening to stories depicting restaurant or airport experiences. Our results reveal a network of brain regions that is sensitive to the shared temporal structure of these naturalistic situations. These regions abstract away from the particular details of each story, activating a representation of the general type of situation being perceived.

Theta oscillations promote temporal sequence learning

Many theoretical models suggest that neural oscillations play a role in learning or retrieval of temporal sequences, but the extent to which oscillations support sequence representation remains unclear. To address this question, we used scalp electroencephalography (EEG) to examine oscillatory activity over learning of different object sequences. Participants made semantic decisions on each object as they were presented in a continuous stream. For three "Consistent" sequences, the order of the objects was always fixed. Activity during Consistent sequences was compared to "Random" sequences that consisted of the same objects presented in a different order on each repetition. Over the course of learning, participants made faster semantic decisions to objects in Consistent, as compared to objects in Random sequences. Thus, participants were able to use sequence knowledge to predict upcoming items in Consistent sequences. EEG analyses revealed decreased oscillatory power in the theta (4-7 Hz) band at frontal sites following decisions about objects in Consistent sequences, as compared with objects in Random sequences. The theta power difference between Consistent and Random only emerged in the second half of the task, as participants were more effectively able to predict items in Consistent sequences. Moreover, we found increases in parieto-occipital alpha (10-13 Hz) and beta (14-28 Hz) power during the pre-response period for objects in Consistent sequences, relative to objects in Random sequences. Linear mixed effects modeling revealed that single trial theta oscillations were related to reaction time for future objects in a sequence, whereas beta and alpha oscillations were only predictive of reaction time on the current trial. These results indicate that theta and alpha/beta activity preferentially relate to future and current events, respectively. More generally our findings highlight the importance of band-specific neural oscillations in the learning of temporal order information.

Predictability matters: role of the hippocampus and prefrontal cortex in disambiguation of overlapping sequences

Previous research has demonstrated that areas in the medial temporal lobe and prefrontal cortex (PFC) show increased activation during retrieval of overlapping sequences. In this study, we designed a task in which degree of overlap varied between conditions in order to parse out the contributions of hippocampal and prefrontal subregions as overlap between associations increased. In the task, participants learned sequential associations consisting of a picture frame, a face within the picture frame, and an outdoor scene. The control condition consisted of a single frame-face-scene sequence. In the low overlap condition, each frame was paired with two faces and two scenes. In the high overlap condition, each frame was paired with four faces and four scenes. In all conditions the correct scene was chosen among four possible scenes and was dependent on the frame and face that preceded the choice point. One day after training, participants were tested on the retrieval of learned sequences during fMRI scanning. Results showed that the middle and posterior hippocampus (HC) was active at times when participants acquired information that increased predictability of the correct response in the overlapping sequences. Activation of dorsolateral PFC occurred at time points when the participant was able to ascertain which set of sequences the correct response belonged to. The ventrolateral PFC was active when inhibition was required, either of irrelevant stimuli or incorrect responses. These results indicate that areas of lateral PFC work in concert with the HC to disambiguate between overlapping sequences and that sequence predictability is key to when specific brain regions become active.

Nonmonotonic recruitment of ventromedial prefrontal cortex during remote memory recall

Systems-level consolidation refers to the time-dependent reorganisation of memory traces in the neocortex, a process in which the ventromedial prefrontal cortex (vmPFC) has been implicated. Capturing the precise temporal evolution of this crucial process in humans has long proved elusive. Here, we used multivariate methods and a longitudinal functional magnetic resonance imaging (fMRI) design to detect, with high granularity, the extent to which autobiographical memories of different ages were represented in vmPFC and how this changed over time. We observed an unexpected time course of vmPFC recruitment during retrieval, rising and falling around an initial peak of 8–12 months, before reengaging for older 2- and 5-year-old memories. This pattern was replicated in 2 independent sets of memories. Moreover, it was further replicated in a follow-up study 8 months later with the same participants and memories, for which the individual memory representations had undergone their hypothesised strengthening or weakening over time. We conclude that the temporal engagement of vmPFC in memory retrieval seems to be nonmonotonic, revealing a complex relationship between systems-level consolidation and prefrontal cortex recruitment that is unaccounted for by current theories.

Our past experiences are captured in autobiographical memories that allow us to recollect events from our lives long after they originally occurred. A part of the brain’s frontal lobe, called the ventromedial prefrontal cortex (vmPFC), is known to be important for supporting autobiographical memories, especially as memories become more remote. The precise temporal profile of the vmPFC’s involvement is unclear, yet this information is vital if we are to understand how memories change over time and the mechanisms involved. In this study, we sought to establish the time course of vmPFC engagement in the recollection of autobiographical memories while participants recalled memories of different ages during functional magnetic resonance imaging (fMRI). Using a method that detects the brain activity patterns associated with individual memories, we found that memory-specific neural patterns in vmPFC became more distinct over the first few months after a memory was formed, but then this initial involvement of vmPFC subsided after 1 year. However, more remote memories (2 years and older) appeared to reengage vmPFC once again. This temporal profile is difficult to accommodate within any single existing theory. Consequently, our results provoke a rethink about how memories evolve over time and the role played by the vmPFC.

Human Episodic Memory Retrieval Is Accompanied by a Neural Contiguity Effect

Cognitive psychologists have long hypothesized that experiences are encoded in a temporal context that changes gradually over time. When an episodic memory is retrieved, the state of context is recovered-a jump back in time. We recorded from single units in the medial temporal lobe of epilepsy patients performing an item recognition task. The population vector changed gradually over minutes during presentation of the list. When a probe from the list was remembered with high confidence, the population vector reinstated the temporal context of the original presentation of that probe during study, a neural contiguity effect that provides a possible mechanism for behavioral contiguity effects. This pattern was only observed for well remembered probes; old probes that were not well remembered showed an anti-contiguity effect. These results constitute the first direct evidence that recovery of an episodic memory in humans is associated with retrieval of a gradually changing state of temporal context, a neural "jump back in time" that parallels the act of remembering.SIGNIFICANCE STATEMENT Episodic memory is the ability to relive a specific experience from one's life. For decades, researchers have hypothesized that, unlike other forms of memory that can be described as simple associations between stimuli, episodic memory depends on the recovery of a neural representation of spatiotemporal context. During study of a sequence of stimuli, the brain state of epilepsy patients changed slowly over at least a minute. When the participant remembered a particular event from the list, this gradually changing state was recovered. This provides direct confirmation of the prediction from computational models of episodic memory. The resolution of this point means that the study of episodic memory can focus on the mechanisms by which this representation of spatiotemporal context is maintained and sometimes recovered.

Frontal Cortex and the Hierarchical Control of Behavior

The frontal lobes are important for cognitive control, yet their functional organization remains controversial. An influential class of theory proposes that the frontal lobes are organized along their rostrocaudal axis to support hierarchical cognitive control. Here, we take an updated look at the literature on hierarchical control, with particular focus on the functional organization of lateral frontal cortex. Our review of the evidence supports neither a unitary model of lateral frontal function nor a unidimensional abstraction gradient. Rather, separate frontal networks interact via local and global hierarchical structure to support diverse task demands.

Time Regained: How the Human Brain Constructs Memory for Time

Life's episodes unfold against a context that changes with time. Recent neuroimaging studies have revealed significant findings about how specific areas of the human brain may support the representation of temporal information in memory. A consistent theme in these studies is that the hippocampus appears to play a central role in representing temporal context, as operationalized in neuroimaging studies of arbitrary lists of items, sequences of items, or meaningful, lifelike events. Additionally, activity in a posterior medial cortical network may reflect the representation of generalized temporal information for meaningful events. The hippocampus, posterior medial network, and other regions-particularly in prefrontal cortex-appear to play complementary roles in memory for temporal context.

Temporal and spatial context in the mind and brain

Theories of episodic memory have long hypothesized that recollection of a specific instance from one's life is mediated by recovery of a neural state of spatiotemporal context. This paper reviews recent theoretical advances in formal models of spatiotemporal context and a growing body of neurophysiological evidence from human imaging studies and animal work that neural populations in the hippocampus and other brain regions support a representation of spatiotemporal context.

On the Integration of Space, Time, and Memory

The hippocampus is famous for mapping locations in spatially organized environments, and several recent studies have shown that hippocampal networks also map moments in temporally organized experiences. Here I consider how space and time are integrated in the representation of memories. The brain pathways for spatial and temporal cognition involve overlapping and interacting systems that converge on the hippocampal region. There is evidence that spatial and temporal aspects of memory are processed somewhat differently in the circuitry of hippocampal subregions but become fully integrated within CA1 neuronal networks as independent, multiplexed representations of space and time. Hippocampal networks also map memories across a broad range of abstract relations among events, suggesting that the findings on spatial and temporal organization reflect a generalized mechanism for organizing memories.

Reading positional codes with fMRI: Problems and solutions

Neural mechanisms which bind items into sequences have been investigated in a large body of research in animal neurophysiology and human neuroimaging. However, a major problem in interpreting this data arises from a fact that several unrelated processes, such as memory load, sensory adaptation, and reward expectation, also change in a consistent manner as the sequence unfolds. In this paper we use computational simulations and data from two fMRI experiments to show that a host of unrelated neural processes can masquerade as sequence representations. We show that dissociating such unrelated processes from a dedicated sequence representation is an especially difficult problem for fMRI data, which is almost exclusively the modality used in human experiments. We suggest that such fMRI results must be treated with caution and in many cases the assumed neural representation might actually reflect unrelated processes.

The retrosplenial cortex and object recency memory in the rat

It has been proposed that the retrosplenial cortex forms part of a ‘where/when’ information network. The present study focussed on the related issue of whether retrosplenial cortex also contributes to ‘what/when’ information, by examining object recency memory. In Experiment 1, rats with retrosplenial lesions were found to be impaired at distinguishing the temporal order of objects presented in a continuous series (‘Within‐Block’ condition). The same lesioned rats could, however, distinguish between objects that had been previously presented in one of two discrete blocks (‘Between‐Block’ condition). Experiment 2 used intact rats to map the expression of the immediate‐early gene c‐fos in retrosplenial cortex following performance of a between‐block, recency discrimination. Recency performance correlated positively with levels of c‐fos expression in both granular and dysgranular retrosplenial cortex (areas 29 and 30). Expression of c‐fos in the granular retrosplenial cortex also correlated with prelimbic cortex and ventral subiculum c‐fos activity, the latter also correlating with recency memory performance. The combined findings from both experiments reveal an involvement of the retrosplenial cortex in temporal order memory, which includes both between‐block and within‐block problems. The current findings also suggest that the rat retrosplenial cortex comprises one of a group of closely interlinked regions that enable recency memory, including the hippocampal formation, medial diencephalon and medial frontal cortex. In view of the well‐established importance of the retrosplenial cortex for spatial learning, the findings support the notion that, with its frontal and hippocampal connections, retrosplenial cortex has a key role for both what/when and where/when information.

A shared representation of order between encoding and recognition in visual short-term memory

Many complex tasks require people to bind individual events into a sequence that can be held in short term memory (STM). For this purpose information about the order of the individual events in the sequence needs to be maintained in an active and accessible form in STM over a period of few seconds. Here we investigated how the temporal order information is shared between the presentation and response phases of an STM task. We trained a classification algorithm on the fMRI activity patterns from the presentation phase of the STM task to predict the order of the items during the subsequent recognition phase. While voxels in a number of brain regions represented positional information during either presentation and recognition phases, only voxels in the lateral prefrontal cortex (PFC) and the anterior temporal lobe (ATL) represented position consistently across task phases. A shared positional code in the ATL might reflect verbal recoding of visual sequences to facilitate the maintenance of order information over several seconds.

Multivoxel Pattern Analysis Reveals 3D Place Information in the Human Hippocampus

The spatial world is three dimensional (3D) and humans and other animals move both horizontally and vertically within it. Extant neuroscientific studies have typically investigated spatial navigation on a horizontal 2D plane, leaving much unknown about how 3D spatial information is represented in the brain. Specifically, horizontal and vertical information may be encoded in the same or different neural structures with equal or unequal sensitivity. Here, we investigated these possibilities using fMRI while participants were passively moved within a 3D lattice structure as if riding a rollercoaster. Multivoxel pattern analysis was used to test for the existence of information relating to where and in which direction participants were heading in this virtual environment. Behaviorally, participants had similarly accurate memory for vertical and horizontal locations and the right anterior hippocampus (HC) expressed place information that was sensitive to changes along both horizontal and vertical axes. This is suggestive of isotropic 3D place encoding. In contrast, participants indicated their heading direction faster and more accurately when they were heading in a tilted-up or tilted-down direction. This direction information was expressed in the right retrosplenial cortex and posterior HC and was only sensitive to vertical pitch, which could reflect the importance of the vertical (gravity) axis as a reference frame. Overall, our findings extend previous knowledge of how we represent the spatial world and navigate within it by taking into account the important third dimension.

SIGNIFICANCE STATEMENT The spatial world is 3D. We can move horizontally across surfaces, but also vertically, going up slopes or stairs. Little is known about how the brain supports representations of 3D space. A key question is whether horizontal and vertical information is equally well represented. Here, we measured fMRI response patterns while participants moved within a virtual 3D environment and found that the anterior hippocampus (HC) expressed location information that was sensitive to the vertical and horizontal axes. In contrast, information about heading direction, found in retrosplenial cortex and posterior HC, favored the vertical axis, perhaps due to gravity effects. These findings provide new insights into how we represent our spatial 3D world and navigate within it.

Learned interval time facilitates associate memory retrieval

The extent to which time is represented in memory remains underinvestigated. We designed a time paired associate task (TPAT) in which participants implicitly learned cue-time-target associations between cue-target pairs and specific cue-target intervals. During subsequent memory testing, participants showed increased accuracy of identifying matching cue-target pairs if the time interval during testing matched the implicitly learned interval. A control experiment showed that participants had no explicit knowledge about the cue-time associations. We suggest that "elapsed time" can act as a temporal mnemonic associate that can facilitate retrieval of events associated in memory.

Recollection-dependent memory for event duration in large-scale spatial navigation

Time and space represent two key aspects of episodic memories, forming the spatiotemporal context of events in a sequence. Little is known, however, about how temporal information, such as the duration and the order of particular events, are encoded into memory, and if it matters whether the memory representation is based on recollection or familiarity. To investigate this issue, we used a real world virtual reality navigation paradigm where periods of navigation were interspersed with pauses of different durations. Crucially, participants were able to reliably distinguish the durations of events that were subjectively “reexperienced” (i.e., recollected), but not of those that were familiar. This effect was not found in temporal order (ordinal) judgments. We also show that the active experience of the passage of time (holding down a key while waiting) moderately enhanced duration memory accuracy. Memory for event duration, therefore, appears to rely on the hippocampally supported ability to recollect or reexperience an event enabling the reinstatement of both its duration and its spatial context, to distinguish it from other events in a sequence. In contrast, ordinal memory appears to rely on familiarity and recollection to a similar extent.

The hippocampus: a special place for time

Many findings have demonstrated that memories of past events are temporally organized. It is well known that the hippocampus is critical for such episodic memories, but, until recently, little was known about the temporal organization of mnemonic representations in the hippocampus. Recent developments in human and animal research have revealed important insights into the role of the hippocampus in learning and retrieving sequences of events. Here, we review these findings, including lesion and single-unit recording studies in rodents, functional magnetic resonance imaging studies in humans, and computational models that link findings from these studies to the anatomy of the hippocampal circuit. The findings converge toward the idea that the hippocampus is essential for learning sequences of events, allowing the brain to distinguish between memories for conceptually similar but temporally distinct episodes, and to associate representations of temporally contiguous, but otherwise unrelated experiences.