Ensembles of human MTL neurons "jump back in time" in response to a repeated stimulus

Re-expression of CA1 and entorhinal activity patterns preserves temporal context memory at long timescales

Converging, cross-species evidence indicates that memory for time is supported by hippocampal area CA1 and entorhinal cortex. However, limited evidence characterizes how these regions preserve temporal memories over long timescales (e.g., months). At long timescales, memoranda may be encountered in multiple temporal contexts, potentially creating interference. Here, using 7T fMRI, we measured CA1 and entorhinal activity patterns as human participants viewed thousands of natural scene images distributed, and repeated, across many months. We show that memory for an image's original temporal context was predicted by the degree to which CA1/entorhinal activity patterns from the first encounter with an image were re-expressed during re-encounters occurring minutes to months later. Critically, temporal memory signals were dissociable from predictors of recognition confidence, which were carried by distinct medial temporal lobe expressions. These findings suggest that CA1 and entorhinal cortex preserve temporal memories across long timescales by coding for and reinstating temporal context information.

Neural temporal context reinstatement of event structure during memory recall

The transformation of experiences into meaningful events and memories is intertwined with the notion of time. Temporal perception can influence, and be influenced by, segmenting continuous experience into meaningful events. Episodic memories formed from these events become associated with temporal information as well. However, it is less clear how temporal perception contributes to structuring events and organizing memory: whether it plays a more active or passive role, and whether this temporal information is encoded initially during perception or influenced by retrieval processes. To address these questions, we examined how event segmentation influences temporal representations during initial perception and memory retrieval, without testing temporal information explicitly. Using a neural measure of temporal context extracted from scalp electroencephalography in human participants (N = 170), we found reduced temporal context similarity between studied items separated by an event boundary when compared to items from the same event. Furthermore, while participants freely recalled list items, neural activity reflected reinstatement of temporal context representations from the study phase, including temporal disruption. A computational model of episodic memory, the context maintenance and retrieval (CMR) model, predicted these results, and made novel predictions regarding the influence of temporal disruption on recall order. These findings implicate the impact of event structure on memory organization via temporal representations, underscoring the role of temporal information in event segmentation and episodic memory. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Cognitive tasks affect the relationship between representational pattern similarity and subsequent item memory in the hippocampus

Episodic memories are records of personally experienced events, coded neurally via the hippocampus and surrounding medial temporal lobe cortex. Information about the neural signal corresponding to a memory representation can be measured in fMRI data when the pattern across voxels is examined. Prior studies have found that similarity in the voxel patterns across repetition of a to-be-remembered stimulus predicts later memory retrieval, but the results are inconsistent across studies. The current study investigates the possibility that cognitive goals (defined here via the task instructions given to participants) during encoding affect the voxel pattern that will later support memory retrieval, and therefore that neural representations cannot be interpreted based on the stimulus alone. The behavioral results showed that exposure to variable cognitive tasks across repetition of events benefited subsequent memory retrieval. Voxel patterns in the hippocampus indicated a significant interaction between cognitive tasks (variable vs. consistent) and memory (remembered vs. forgotten) such that reduced voxel pattern similarity for repeated events with variable cognitive tasks, but not consistent cognitive tasks, supported later memory success. There was no significant interaction in neural pattern similarity between cognitive tasks and memory success in medial temporal cortices or lateral occipital cortex. Instead, higher similarity in voxel patterns in right medial temporal cortices was associated with later memory retrieval, regardless of cognitive task. In conclusion, we found that the relationship between pattern similarity across repeated encoding and memory success in the hippocampus (but not medial temporal lobe cortex) changes when the cognitive task during encoding does or does not vary across repetitions of the event.

Memory consolidation during sleep involves context reinstatement in humans

New memories are not quarantined from each other when first encoded; rather, they are interlinked with memories that were encoded in temporal proximity or that share semantic features. By selectively biasing memory processing during sleep, here we test whether context influences sleep consolidation. Participants first formed 18 idiosyncratic narratives, each linking four objects together. Before sleep, they also memorized an on-screen position for each object. During sleep, 12 object-specific sounds were unobtrusively presented, thereby cuing the corresponding spatial memories and impacting spatial recall as a function of initial memory strength. As hypothesized, we find that recall for non-cued objects contextually linked with cued objects also changed. Post-cue electrophysiological responses suggest that activity in the sigma band supports context reinstatement and predicts context-related memory benefits. Concurrently, context-specific electrophysiological activity patterns emerge during sleep. We conclude that reactivation of individual memories during sleep evokes reinstatement of their context, thereby impacting consolidation of associated knowledge.

Hippocampal ripples signal contextually mediated episodic recall

High-frequency oscillatory events, termed ripples, represent synchrony of neural activity in the brain. Recent evidence suggests that medial temporal lobe (MTL) ripples support memory retrieval. However, it is unclear if ripples signal the reinstatement of episodic memories. Analyzing electrophysiological MTL recordings from 245 neurosurgical participants performing episodic recall tasks, we find that the rate of hippocampal ripples rises just prior to the free recall of recently formed memories. This prerecall ripple effect (PRE) is stronger in the CA1 and CA3/dentate gyrus (CA3/DG) subfields of the hippocampus than the neighboring MTL regions entorhinal and parahippocampal cortex. PRE is also stronger prior to the retrieval of temporally and semantically clustered, as compared with unclustered, recalls, indicating the involvement of ripples in contextual reinstatement, which is a hallmark of episodic memory.

The neural bases for timing of durations

Durations are defined by a beginning and an end, and a major distinction is drawn between durations that start in the present and end in the future ('prospective timing') and durations that start in the past and end either in the past or the present ('retrospective timing'). Different psychological processes are thought to be engaged in each of these cases. The former is thought to engage a clock-like mechanism that accurately tracks the continuing passage of time, whereas the latter is thought to engage a reconstructive process that utilizes both temporal and non-temporal information from the memory of past events. We propose that, from a biological perspective, these two forms of duration 'estimation' are supported by computational processes that are both reliant on population state dynamics but are nevertheless distinct. Prospective timing is effectively carried out in a single step where the ongoing dynamics of population activity directly serve as the computation of duration, whereas retrospective timing is carried out in two steps: the initial generation of population state dynamics through the process of event segmentation and the subsequent computation of duration utilizing the memory of those dynamics.

Neurons in the human medial temporal lobe track multiple temporal contexts during episodic memory processing

Episodic memory requires associating items with temporal context, a process for which the medial temporal lobe (MTL) is critical. This study uses recordings from 27 human subjects who were undergoing surgical intervention for intractable epilepsy. These same data were also utilized in Umbach et al. (2020). We identify 103 memory-sensitive neurons in the hippocampus and entorhinal cortex, whose firing rates predicted successful episodic memory encoding as subjects performed a verbal free recall task. These neurons exhibit important properties. First, as predicted from the temporal context model, they demonstrate reinstatement of firing patterns observed during encoding at the time of retrieval. The magnitude of reinstatement predicted the tendency of subjects to cluster retrieved memory items according to input serial position. Also, we found that spiking activity of these neurons was locked to the phase of hippocampal theta oscillations, but that the mean phase of spiking shifted between memory encoding versus retrieval. This unique observation is consistent with predictions of the “Separate Phases at Encoding And Retrieval (SPEAR)” model. Together, the properties we identify for memory-sensitive neurons characterize direct electrophysiological mechanisms for the representation of contextual information in the human MTL.

Human Hippocampal Neurons Track Moments in a Sequence of Events

An indispensable feature of episodic memory is our ability to temporally piece together different elements of an experience into a coherent memory. Hippocampal time cells-neurons that represent temporal information-may play a critical role in this process. Although these cells have been repeatedly found in rodents, it is still unclear to what extent similar temporal selectivity exists in the human hippocampus. Here, we show that temporal context modulates the firing activity of human hippocampal neurons during structured temporal experiences. We recorded neuronal activity in the human brain while patients of either sex learned predictable sequences of pictures. We report that human time cells fire at successive moments in this task. Furthermore, time cells also signaled inherently changing temporal contexts during empty 10 s gap periods between trials while participants waited for the task to resume. Finally, population activity allowed for decoding temporal epoch identity, both during sequence learning and during the gap periods. These findings suggest that human hippocampal neurons could play an essential role in temporally organizing distinct moments of an experience in episodic memory.SIGNIFICANCE STATEMENT Episodic memory refers to our ability to remember the what, where, and when of a past experience. Representing time is an important component of this form of memory. Here, we show that neurons in the human hippocampus represent temporal information. This temporal signature was observed both when participants were actively engaged in a memory task, as well as during 10-s-long gaps when they were asked to wait before performing the task. Furthermore, the activity of the population of hippocampal cells allowed for decoding one temporal epoch from another. These results suggest a robust representation of time in the human hippocampus.

Geometric models reveal behavioural and neural signatures of transforming experiences into memories

How do we preserve and distort our ongoing experiences when encoding them into episodic memories? The mental contexts in which we interpret experiences are often person-specific, even when the experiences themselves are shared. Here we develop a geometric framework for mathematically characterizing the subjective conceptual content of dynamic naturalistic experiences. We model experiences and memories as trajectories through word-embedding spaces whose coordinates reflect the universe of thoughts under consideration. Memory encoding can then be modelled as geometrically preserving or distorting the 'shape' of the original experience. We applied our approach to data collected as participants watched and verbally recounted a television episode while undergoing functional neuroimaging. Participants' recountings preserved coarse spatial properties (essential narrative elements) but not fine spatial scale (low-level) details of the episode's trajectory. We also identified networks of brain structures sensitive to these trajectory shapes.

The Sync-Fire/deSync model: Modelling the reactivation of dynamic memories from cortical alpha oscillations

We propose a neural network model to explore how humans can learn and accurately retrieve temporal sequences, such as melodies, movies, or other dynamic content. We identify target memories by their neural oscillatory signatures, as shown in recent human episodic memory paradigms. Our model comprises three plausible components for the binding of temporal content, where each component imposes unique limitations on the encoding and representation of that content. A cortical component actively represents sequences through the disruption of an intrinsically generated alpha rhythm, where a desynchronisation marks information-rich operations as the literature predicts. A binding component converts each event into a discrete index, enabling repetitions through a sparse encoding of events. A timing component - consisting of an oscillatory "ticking clock" made up of hierarchical synfire chains - discretely indexes a moment in time. By encoding the absolute timing between discretised events, we show how one can use cortical desynchronisations to dynamically detect unique temporal signatures as they are reactivated in the brain. We validate this model by simulating a series of events where sequences are uniquely identifiable by analysing phasic information, as several recent EEG/MEG studies have shown. As such, we show how one can encode and retrieve complete episodic memories where the quality of such memories is modulated by the following: alpha gate keepers to content representation; binding limitations that induce a blink in temporal perception; and nested oscillations that provide preferential learning phases in order to temporally sequence events.

Medial prefrontal cortex and hippocampal activity differentially contribute to ordinal and temporal context retrieval during sequence memory

Remembering sequences of events defines episodic memory, but retrieval can be driven by both ordinality and temporal contexts. Whether these modes of retrieval operate at the same time or not remains unclear. Theoretically, medial prefrontal cortex (mPFC) confers ordinality, while the hippocampus (HC) associates events in gradually changing temporal contexts. Here, we looked for evidence of each with BOLD fMRI in a sequence task that taxes both retrieval modes. To test ordinal modes, items were transferred between sequences but retained their position (e.g., AB3). Ordinal modes activated mPFC, but not HC. To test temporal contexts, we examined items that skipped ahead across lag distances (e.g., ABD). HC, but not mPFC, tracked temporal contexts. There was a mPFC and HC by retrieval mode interaction. These current results suggest that the mPFC and HC are concurrently engaged in different retrieval modes in support of remembering when an event occurred.

Time cells in the human hippocampus and entorhinal cortex support episodic memory

The organization of temporal information is critical for the encoding and retrieval of episodic memories. In the rodent hippocampus and entorhinal cortex, evidence accumulated over the last decade suggests that populations of "time cells" in the hippocampus encode temporal information. We identify time cells in humans using intracranial microelectrode recordings obtained from 27 human epilepsy patients who performed an episodic memory task. We show that time cell activity predicts the temporal organization of retrieved memory items. We also uncover evidence of ramping cell activity in humans, which represents a complementary type of temporal information. These findings establish a cellular mechanism for the representation of temporal information in the human brain needed to form episodic memories.

Temporal context guides visual exploration during scene recognition

Memories for episodes are temporally structured. Cognitive models derived from list-learning experiments attribute this structure to the retrieval of temporal context information that indicates when a memory occurred. These models predict key features of memory recall, such as the strong tendency to retrieve studied items in the order in which they were first encountered. Can such models explain ecological memory behaviors, such as eye movements during encoding and retrieval of complex visual stimuli? We tested predictions from retrieved-context models using three data sets involving recognition memory and free viewing of complex scenes. Subjects reinstated sequences of eye movements from one scene-viewing episode to the next. Moreover, sequence reinstatement decayed over time and was associated with successful memory. We observed memory-driven reinstatement even after accounting for intrinsic scene properties that produced consistent eye movements. These findings confirm predictions of retrieved-context models, suggesting retrieval of temporal context influences complex behaviors generated during naturalistic memory experiences. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The Generation of Time in the Hippocampal Memory System

We propose that ramping time cells in the lateral entorhinal cortex can be produced by synaptic adaptation and demonstrate this in an integrate-and-fire attractor network model. We propose that competitive networks in the hippocampal system can convert these entorhinal ramping cells into hippocampal time cells and demonstrate this in a competitive network. We propose that this conversion is necessary to provide orthogonal hippocampal time representations to encode the temporal sequence of events in hippocampal episodic memory, and we support that with analytic arguments. We demonstrate that this processing can produce hippocampal neuronal ensembles that not only show replay of the sequence later on, but can also do this in reverse order in reverse replay. This research addresses a major issue in neuroscience: the mechanisms by which time is encoded in the brain and how the time representations are then useful in the hippocampal memory of events and their order.

Sequence Memory in the Hippocampal-Entorhinal Region

Episodic memories are constructed from sequences of events. When recalling such a memory, we not only recall individual events, but we also retrieve information about how the sequence of events unfolded. Here, we focus on the role of the hippocampal-entorhinal region in processing and remembering sequences of events, which are thought to be stored in relational networks. We summarize evidence that temporal relations are a central organizational principle for memories in the hippocampus. Importantly, we incorporate novel insights from recent studies about the role of the adjacent entorhinal cortex in sequence memory. In rodents, the lateral entorhinal subregion carries temporal information during ongoing behavior. The human homologue is recruited during memory recall where its representations reflect the temporal relationships between events encountered in a sequence. We further introduce the idea that the hippocampal-entorhinal region might enable temporal scaling of sequence representations. Flexible changes of sequence progression speed could underlie the traversal of episodic memories and mental simulations at different paces. In conclusion, we describe how the entorhinal cortex and hippocampus contribute to remembering event sequences-a core component of episodic memory.

Electrophysiological signatures of memory reactivation in humans

The reactivation of neural activity that was present during the encoding of an event is assumed to be essential for human episodic memory retrieval and the consolidation of memories during sleep. Pioneering animal work has already established a crucial role of memory reactivation to prepare and guide behaviour. Research in humans is now delineating the neural processes involved in memory reactivation during both wakefulness and sleep as well as their functional significance. Focusing on the electrophysiological signatures of memory reactivation in humans during both memory retrieval and sleep-related consolidation, this review provides an overview of the state of the art in the field. We outline recent advances, methodological developments and open questions and specifically highlight commonalities and differences in the neuronal signatures of memory reactivation during the states of wakefulness and sleep.

This article is part of the Theo Murphy meeting issue ‘Memory reactivation: replaying events past, present and future’.

The hippocampus contributes to temporal duration memory in the context of event sequences: A cross-species perspective

Although a large body of research has implicated the hippocampus in the processing of memory for temporal duration, there is an exigent degree of inconsistency across studies that obfuscates the precise contributions of this structure. To shed light on this issue, the present review article surveys both historical and recent cross-species evidence emanating from a wide variety of experimental paradigms, identifying areas of convergence and divergence. We suggest that while factors such as time-scale (e.g. the length of durations involved) and the nature of memory processing (e.g. prospective vs. retrospective memory) are very helpful in the interpretation of existing data, an additional important consideration is the context in which the duration information is experienced and processed, with the hippocampus being preferentially involved in memory for durations that are embedded within a sequence of events. We consider the mechanisms that may underpin temporal duration memory and how the same mechanisms may contribute to memory for other aspects of event sequences such as temporal order.

Mapping sequence structure in the human lateral entorhinal cortex

Remembering event sequences is central to episodic memory and presumably supported by the hippocampal-entorhinal region. We previously demonstrated that the hippocampus maps spatial and temporal distances between events encountered along a route through a virtual city (Deuker et al., 2016), but the content of entorhinal mnemonic representations remains unclear. Here, we demonstrate that multi-voxel representations in the anterior-lateral entorhinal cortex (alEC) - the human homologue of the rodent lateral entorhinal cortex - specifically reflect the temporal event structure after learning. Holistic representations of the sequence structure related to memory recall and the timeline of events could be reconstructed from entorhinal multi-voxel patterns. Our findings demonstrate representations of temporal structure in the alEC; dovetailing with temporal information carried by population signals in the lateral entorhinal cortex of navigating rodents and alEC activations during temporal memory retrieval. Our results provide novel evidence for the role of the alEC in representing time for episodic memory.

Episodic memory: Neuronal codes for what, where, and when

Episodic memory is defined as the ability to recall events in a spatiotemporal context. Formation of such memories is critically dependent on the hippocampal formation and its inputs from the entorhinal cortex. To be able to support the formation of episodic memories, entorhinal cortex and hippocampal formation should contain a neuronal code that follows several requirements. First, the code should include information about position of the agent ("where"), sequence of events ("when"), and the content of the experience itself ("what"). Second, the code should arise instantly thereby being able to support memory formation of one-shot experiences. For successful encoding and to avoid interference between memories during recall, variations in location, time, or in content of experience should result in unique ensemble activity. Finally, the code should capture several different resolutions of experience so that the necessary details relevant for future memory-based predictions will be stored. We review how neuronal codes in entorhinal cortex and hippocampus follow these requirements and argue that during formation of episodic memories entorhinal cortex provides hippocampus with instant information about ongoing experience. Such information originates from (a) spatially modulated neurons in medial entorhinal cortex, including grid cells, which provide a stable and universal positional metric of the environment; (b) a continuously varying signal in lateral entorhinal cortex providing a code for the temporal progression of events; and (c) entorhinal neurons coding the content of experiences exemplified by object-coding and odor-selective neurons. During formation of episodic memories, information from these systems are thought to be encoded as unique sequential ensemble activity in hippocampus, thereby encoding associations between the content of an event and its spatial and temporal contexts. Upon exposure to parts of the encoded stimuli, activity in these ensembles can be reinstated, leading to reactivation of the encoded activity pattern and memory recollection.

A contextual binding theory of episodic memory: systems consolidation reconsidered

Episodic memory reflects the ability to recollect the temporal and spatial context of past experiences. Episodic memories depend on the hippocampus but have been proposed to undergo rapid forgetting unless consolidated during offline periods such as sleep to neocortical areas for long-term storage. Here, we propose an alternative to this standard systems consolidation theory (SSCT) - a contextual binding account - in which the hippocampus binds item-related and context-related information. We compare these accounts in light of behavioural, lesion, neuroimaging and sleep studies of episodic memory and contend that forgetting is largely due to contextual interference, episodic memory remains dependent on the hippocampus across time, contextual drift produces post-encoding activity and sleep benefits memory by reducing contextual interference.

Testing Models of Human Declarative Memory at the Single-Neuron Level

Deciphering the mechanisms of declarative memory is a major goal of neuroscience. While much theoretical progress has been made, it has proven difficult to experimentally verify key predictions of some foundational models of memory. Recently, single-neuron recordings in human patients have started to provide direct experimental verification of some theories, including mnemonic evidence accumulation, balance-of-evidence for confidence judgments, sparse coding, contextual reinstatement, and the ventral tegmental area (VTA)-hippocampus loop model. Here, we summarize the cell types that have been described in the medial temporal lobe and posterior parietal cortex, discuss their properties, and reflect on how these findings inform theoretical work. This body of work exemplifies the scientific power of a synergistic combination of modeling and human single-neuron recordings to advance cognitive neuroscience.

Hippocampal contributions to serial-order memory

Our memories form a record not only of our experiences, but also of their temporal structure. Although memory for the temporal structure of experience likely relies on multiple neural systems, numerous studies have implicated the hippocampus in the encoding and retrieval of temporal information. This review evaluates the literature on hippocampal contributions to human serial-order memory from the perspective of three cognitive theories: associative chaining theory, positional-coding theory and retrieved-context theory. Evaluating neural findings through the lens of cognitive theories enables us to draw more incisive conclusions about the relations between brain and behavior.

The storage and recall of memories in the hippocampo-cortical system

A quantitative computational theory of the operation of the hippocampus as an episodic memory system is described. The CA3 system operates as a single attractor or autoassociation network (1) to enable rapid one-trial associations between any spatial location (place in rodents or spatial view in primates) and an object or reward and (2) to provide for completion of the whole memory during recall from any part. The theory is extended to associations between time and object or reward to implement temporal order memory, which is also important in episodic memory. The dentate gyrus performs pattern separation by competitive learning to create sparse representations producing, for example, neurons with place-like fields from entorhinal cortex grid cells. The dentate granule cells generate, by the very small number of mossy fibre connections to CA3, a randomizing pattern separation effect that is important during learning but not recall and that separates out the patterns represented by CA3 firing as being very different from each other. This is optimal for an unstructured episodic memory system in which each memory must be kept distinct from other memories. The direct perforant path input to CA3 is quantitatively appropriate for providing the cue for recall in CA3 but not for learning. The CA1 recodes information from CA3 to set up associatively learned backprojections to the neocortex to allow the subsequent retrieval of information to the neocortex, giving a quantitative account of the large number of hippocampo-neocortical and neocortical-neocortical backprojections. Tests of the theory including hippocampal subregion analyses and hippocampal NMDA receptor knockouts are described and support the theory.

Electrical Stimulation in Hippocampus and Entorhinal Cortex Impairs Spatial and Temporal Memory

The medial temporal lobe (MTL) is widely implicated in supporting episodic memory and navigation, but its precise functional role in organizing memory across time and space remains elusive. Here we examine the specific cognitive processes implemented by MTL structures (hippocampus and entorhinal cortex) to organize memory by using electrical brain stimulation, leveraging its ability to establish causal links between brain regions and features of behavior. We studied neurosurgical patients of both sexes who performed spatial-navigation and verbal-episodic memory tasks while brain stimulation was applied in various regions during learning. During the verbal memory task, stimulation in the MTL disrupted the temporal organization of encoded memories such that items learned with stimulation tended to be recalled in a more randomized order. During the spatial task, MTL stimulation impaired subjects' abilities to remember items located far away from boundaries. These stimulation effects were specific to the MTL. Our findings thus provide the first causal demonstration in humans of the specific memory processes that are performed by the MTL to encode when and where events occurred.SIGNIFICANCE STATEMENT Numerous studies have implicated the medial temporal lobe (MTL) in encoding spatial and temporal memories, but they have not been able to causally demonstrate the nature of the cognitive processes by which this occurs in real-time. Electrical brain stimulation is able to demonstrate causal links between a brain region and a given function with high temporal precision. By examining behavior in a memory task as subjects received MTL stimulation, we provide the first causal evidence demonstrating the role of the MTL in organizing the spatial and temporal aspects of episodic memory.

Human Episodic Memory Retrieval Is Accompanied by a Neural Contiguity Effect

Cognitive psychologists have long hypothesized that experiences are encoded in a temporal context that changes gradually over time. When an episodic memory is retrieved, the state of context is recovered-a jump back in time. We recorded from single units in the medial temporal lobe of epilepsy patients performing an item recognition task. The population vector changed gradually over minutes during presentation of the list. When a probe from the list was remembered with high confidence, the population vector reinstated the temporal context of the original presentation of that probe during study, a neural contiguity effect that provides a possible mechanism for behavioral contiguity effects. This pattern was only observed for well remembered probes; old probes that were not well remembered showed an anti-contiguity effect. These results constitute the first direct evidence that recovery of an episodic memory in humans is associated with retrieval of a gradually changing state of temporal context, a neural "jump back in time" that parallels the act of remembering.SIGNIFICANCE STATEMENT Episodic memory is the ability to relive a specific experience from one's life. For decades, researchers have hypothesized that, unlike other forms of memory that can be described as simple associations between stimuli, episodic memory depends on the recovery of a neural representation of spatiotemporal context. During study of a sequence of stimuli, the brain state of epilepsy patients changed slowly over at least a minute. When the participant remembered a particular event from the list, this gradually changing state was recovered. This provides direct confirmation of the prediction from computational models of episodic memory. The resolution of this point means that the study of episodic memory can focus on the mechanisms by which this representation of spatiotemporal context is maintained and sometimes recovered.

Memory as Perception of the Past: Compressed Time inMind and Brain

In the visual system retinal space is compressed such that acuity decreases further from the fovea. Different forms of memory may rely on a compressed representation of time, manifested as decreased accuracy for events that happened further in the past. Neurophysiologically, "time cells" show receptive fields in time. Analogous to the compression of visual space, time cells show less acuity for events further in the past. Behavioral evidence suggests memory can be accessed by scanning a compressed temporal representation, analogous to visual search. This suggests a common computational language for visual attention and memory retrieval. In this view, time functions like a scaffolding that organizes memories in much the same way that retinal space functions like a scaffolding for visual perception.

Temporal and spatial context in the mind and brain

Theories of episodic memory have long hypothesized that recollection of a specific instance from one's life is mediated by recovery of a neural state of spatiotemporal context. This paper reviews recent theoretical advances in formal models of spatiotemporal context and a growing body of neurophysiological evidence from human imaging studies and animal work that neural populations in the hippocampus and other brain regions support a representation of spatiotemporal context.

Human Cortical Neurons in the Anterior Temporal Lobe Reinstate Spiking Activity during Verbal Memory Retrieval

When we recall an experience, we rely upon the associations that we formed during the experience, such as those among objects, time, and place [1]. These associations are better remembered when they are familiar and draw upon generalized knowledge, suggesting that we use semantic memory in the service of episodic memory [2, 3]. Moreover, converging evidence suggests that episodic memory retrieval involves the reinstatement of neural activity that was present when we first experienced the event. Therefore, we hypothesized that retrieving associations should also reinstate the neural activity responsible for semantic processing. Indeed, previous studies have suggested that verbal memory retrieval leads to the reinstatement of activity across regions of the brain that include the distributed semantic processing network [4-6], but it is unknown whether and how individual neurons in the human cortex participate in the reinstatement of semantic representations. Recent advances using high-density microelectrode arrays (MEAs) have allowed clinicians to record from populations of neurons in the human cortex [7, 8]. Here we used MEAs to record neuronal spiking activity in the human middle temporal gyrus (MTG), a cortical region supporting the semantic representation of words [9-11], as participants performed a verbal paired-associates task. We provide novel evidence that population spiking activity in the MTG forms distinct representations of semantic concepts and that these representations are reinstated during the retrieval of those words.

Cued Memory Retrieval Exhibits Reinstatement of High Gamma Power on a Faster Timescale in the Left Temporal Lobe and Prefrontal Cortex

Converging evidence suggests that reinstatement of neural activity underlies our ability to successfully retrieve memories. However, the temporal dynamics of reinstatement in the human cortex remain poorly understood. One possibility is that neural activity during memory retrieval, like replay of spiking neurons in the hippocampus, occurs at a faster timescale than during encoding. We tested this hypothesis in 34 participants who performed a verbal episodic memory task while we recorded high gamma (62-100 Hz) activity from subdural electrodes implanted for seizure monitoring. We show that reinstatement of distributed patterns of high gamma activity occurs faster than during encoding. Using a time-warping algorithm, we quantify the timescale of the reinstatement and identify brain regions that show significant timescale differences between encoding and retrieval. Our data suggest that temporally compressed reinstatement of cortical activity is a feature of cued memory retrieval.SIGNIFICANCE STATEMENT We show that cued memory retrieval reinstates neural activity on a faster timescale than was present during encoding. Our data therefore provide a link between reinstatement of neural activity in the cortex and spontaneous replay of cortical and hippocampal spiking activity, which also exhibits temporal compression, and suggest that temporal compression may be a universal feature of memory retrieval.

Lingering representations of stimuli influence recall organization

Several prominent theories posit that information about recent experiences lingers in the brain and organizes memories for current experiences, by forming a temporal context that is linked to those memories at encoding. According to these theories, if the thoughts preceding an experience X resemble the thoughts preceding an experience Y, then X and Y should show an elevated probability of being recalled together. We tested this prediction by using multi-voxel pattern analysis (MVPA) of fMRI data to measure neural evidence for lingering processing of preceding stimuli. As predicted, memories encoded with similar lingering thoughts about the category of preceding stimuli were more likely to be recalled together. Our results demonstrate that the "fading embers" of previous stimuli help to organize recall, confirming a key prediction of computational models of episodic memory.

Task-specific Aspects of Goal-directed Word Generation Identified via Simultaneous EEG–fMRI

Generating words according to a given rule relies on retrieval-related search and postretrieval control processes. Using fMRI, we recently characterized neural patterns of word generation in response to episodic, semantic, and phonemic cues by comparing free recall of wordlists, category fluency, and letter fluency [Shapira-Lichter, I., Oren, N., Jacob, Y., Gruberger, M., & Hendler, T. Portraying the unique contribution of the default mode network to internally driven mnemonic processes. Proceedings of the National Academy of Sciences, U.S.A., 110, 4950–4955, 2013]. Distinct selectivity for each condition was evident, representing discrete aspects of word generation-related memory retrieval. For example, the precuneus, implicated in processing spatiotemporal information, emerged as a key contributor to the episodic condition, which uniquely requires this information. Gamma band is known to play a central role in memory, and increased gamma power has been observed before word generation. Yet, gamma modulation in response to task demands has not been investigated. To capture the task-specific modulation of gamma power, we analyzed the EEG data recorded simultaneously with the aforementioned fMRI, focusing on the activity locked to and immediately preceding word articulation. Transient increases in gamma power were identified in a parietal electrode immediately before episodic and semantic word generation, however, within a different time frame relative to articulation. Gamma increases were followed by an alpha-theta decrease in the episodic condition, a gamma decrease in the semantic condition. This pattern indicates a task-specific modulation of the gamma signal corresponding to the specific demands of each word generation task. The gamma power and fMRI signal from the precuneus were correlated during the episodic condition, implying the existence of a common cognitive construct uniquely required for this task, possibly the reactivation or processing of spatiotemporal information.

Distinct neural mechanisms for remembering when an event occurred

Events are often remembered as having occurred in a specific order, but almost nothing is known about how the brain encodes this temporal information. It is commonly assumed that temporal information is encoded via a single mechanism, based either on the temporal context in which the event occurred or inferred from the strength of the memory trace itself. By analyzing time-dependent changes in activity patterns, we show that the distinctiveness of contextual representations in the hippocampus and anterior and medial prefrontal cortex was associated with accurate recency memory. In contrast, overall activation in the perirhinal and lateral prefrontal cortices predicted whether an object would be judged more recent, regardless of accuracy. These results demonstrate that temporal information was encoded through at least two complementary neural mechanisms.

Decoding Episodic Retrieval Processes: Frontoparietal and Medial Temporal Lobe Contributions to Free Recall

Neuroimaging studies of recognition memory have identified distinct patterns of cortical activity associated with two sets of cognitive processes: Recollective processes supporting retrieval of information specifying a probe item's original source are associated with the posterior hippocampus, ventral posterior parietal cortex, and medial pFC. Familiarity processes supporting the correct identification of previously studied probes (in the absence of a recollective response) are associated with activity in anterior medial temporal lobe (MTL) structures including the perirhinal cortex and anterior hippocampus, in addition to lateral prefrontal and dorsal posterior parietal cortex. Here, we address an open question in the cognitive neuroscientific literature: To what extent are these same neurocognitive processes engaged during an internally directed memory search task like free recall? We recorded fMRI activity while participants performed a series of free recall and source recognition trials, and we used a combination of univariate and multivariate analysis techniques to compare neural activation profiles across the two tasks. Univariate analyses showed that posterior MTL regions were commonly associated with recollective processes during source recognition and with free recall responses. Prefrontal and posterior parietal regions were commonly associated with familiarity processes and free recall responses, whereas anterior MTL regions were only associated with familiarity processes during recognition. In contrast with the univariate results, free recall activity patterns characterized using multivariate pattern analysis did not reliably match the neural patterns associated with recollective processes. However, these free recall patterns did reliably match patterns associated with familiarity processes, supporting theories of memory in which common cognitive mechanisms support both item recognition and free recall.

Time and space in the hippocampus

It has been hypothesized that one of the functions of the hippocampus is to enable the learning of relationships between different stimuli experienced in the environment. These relationships might be spatial ("the bathroom is about 5m down the hall from the bedroom") or temporal ("the coffee is ready about 3 min after the button was pressed"). Critically, these spatial and temporal relationships may exist on a variety of scales from a few hundred milliseconds up to minutes. In order to learn consistent relationships between stimuli separated by a variety of spatial and temporal scales using synaptic plasticity that has a fixed temporal window extending at most a few hundred milliseconds, information about the spatial and temporal relationships of distant stimuli must be available to the hippocampus in the present. Hippocampal place cells and time cells seem well suited to represent the spatial and temporal locations of distant stimuli in order to support learning of these relationships. We review a recent computational hypothesis that can be used to construct both spatial and temporal relationships. We suggest that there is a deep computational connection between spatial and temporal coding in the hippocampus and that both serve the overarching function of learning relationships between stimuli-constructing a "memory space." This article is part of a Special Issue entitled SI: Brain and Memory.

A temporal context repetition effect in rats during a novel object recognition memory task

Recent research in humans has used formal models of temporal context, broadly defined as a lingering representation of recent experience, to explain a wide array of recall and recognition memory phenomena. One difficulty in extending this work to studies of experimental animals has been the challenge of developing a task to test temporal context effects on performance in rodents. The current study presents results from a novel object recognition memory paradigm that was adapted from a task used in humans and demonstrates a temporal context repetition effect in rats. Specifically, the findings indicate that repeating the first two objects from a once-encountered sequence of three objects incidentally cues memory for the third object, even in its absence. These results reveal that temporal context influences item memory in rats similar to the manner in which it influences memory in humans and also highlight a new task for future studies of temporal context in experimental animals.

Neural activity in the medial temporal lobe reveals the fidelity of mental time travel

Neural circuitry in the medial temporal lobe (MTL) is critically involved in mental time travel, which involves the vivid retrieval of the details of past experience. Neuroscientific theories propose that the MTL supports memory of the past by retrieving previously encoded episodic information, as well as by reactivating a temporal code specifying the position of a particular event within an episode. However, the neural computations supporting these abilities are underspecified. To test hypotheses regarding the computational mechanisms supported by different MTL subregions during mental time travel, we developed a computational model that linked a blood oxygenation level-dependent signal to cognitive operations, allowing us to predict human performance in a memory search task. Activity in the posterior MTL, including parahippocampal cortex, reflected how strongly one reactivates the temporal context of a retrieved memory, allowing the model to predict whether the next memory will correspond to a nearby moment in the study episode. A signal in the anterior MTL, including perirhinal cortex, indicated the successful retrieval of list items, without providing information regarding temporal organization. A hippocampal signal reflected both processes, consistent with theories that this region binds item and context information together to form episodic memories. These findings provide evidence for modern theories that describe complementary roles of the hippocampus and surrounding parahippocampal and perirhinal cortices during the retrieval of episodic memories, shaping how humans revisit the past.

Memory Without Consolidation: Temporal Distinctiveness Explains Retroactive Interference

Is consolidation needed to account for retroactive interference in free recall? Interfering mental activity during the retention interval of a memory task impairs performance, in particular if the interference occurs in temporal proximity to the encoding of the to-be-remembered (TBR) information. There are at least two rival theoretical accounts of this temporal gradient of retroactive interference. The cognitive neuroscience literature has suggested neural consolidation is a pivotal factor determining item recall. According to this account, interfering activity interrupts consolidation processes that would otherwise stabilize the memory representations of TBR items post-encoding. Temporal distinctiveness theory, by contrast, proposes that the retrievability of items depends on their isolation in psychological time. According to this theory, information processed after the encoding of TBR material will reduce the temporal distinctiveness of the TBR information. To test between these accounts, implementations of consolidation were added to the SIMPLE model of memory and learning. We report data from two experiments utilizing a two-list free recall paradigm. Modeling results imply that SIMPLE was able to model the data and did not benefit from the addition of consolidation. It is concluded that the temporal gradient of retroactive interference cannot be taken as evidence for memory consolidation.

Reinstatement of distributed cortical oscillations occurs with precise spatiotemporal dynamics during successful memory retrieval

Reinstatement of neural activity is hypothesized to underlie our ability to mentally travel back in time to recover the context of a previous experience. We used intracranial recordings to directly examine the precise spatiotemporal extent of neural reinstatement as 32 participants with electrodes placed for seizure monitoring performed a paired-associates episodic verbal memory task. By cueing recall, we were able to compare reinstatement during correct and incorrect trials, and found that successful retrieval occurs with reinstatement of a gradually changing neural signal present during encoding. We examined reinstatement in individual frequency bands and individual electrodes and found that neural reinstatement was largely mediated by temporal lobe theta and high-gamma frequencies. Leveraging the high temporal precision afforded by intracranial recordings, our data demonstrate that high-gamma activity associated with reinstatement preceded theta activity during encoding, but during retrieval this difference in timing between frequency bands was absent. Our results build upon previous studies to provide direct evidence that successful retrieval involves the reinstatement of a temporal context, and that such reinstatement occurs with precise spatiotemporal dynamics.

Time cells in the hippocampus: a new dimension for mapping memories

Recent studies have revealed the existence of hippocampal neurons that fire at successive moments in temporally structured experiences. Several studies have shown that such temporal coding is not attributable to external events, specific behaviours or spatial dimensions of an experience. Instead, these cells represent the flow of time in specific memories and have therefore been dubbed 'time cells'. The firing properties of time cells parallel those of hippocampal place cells; time cells thus provide an additional dimension that is integrated with spatial mapping. The robust representation of both time and space in the hippocampus suggests a fundamental mechanism for organizing the elements of experience into coherent memories.

A unified mathematical framework for coding time, space, and sequences in the hippocampal region

The medial temporal lobe (MTL) is believed to support episodic memory, vivid recollection of a specific event situated in a particular place at a particular time. There is ample neurophysiological evidence that the MTL computes location in allocentric space and more recent evidence that the MTL also codes for time. Space and time represent a similar computational challenge; both are variables that cannot be simply calculated from the immediately available sensory information. We introduce a simple mathematical framework that computes functions of both spatial location and time as special cases of a more general computation. In this framework, experience unfolding in time is encoded via a set of leaky integrators. These leaky integrators encode the Laplace transform of their input. The information contained in the transform can be recovered using an approximation to the inverse Laplace transform. In the temporal domain, the resulting representation reconstructs the temporal history. By integrating movements, the equations give rise to a representation of the path taken to arrive at the present location. By modulating the transform with information about allocentric velocity, the equations code for position of a landmark. Simulated cells show a close correspondence to neurons observed in various regions for all three cases. In the temporal domain, novel secondary analyses of hippocampal time cells verified several qualitative predictions of the model. An integrated representation of spatiotemporal context can be computed by taking conjunctions of these elemental inputs, leading to a correspondence with conjunctive neural representations observed in dorsal CA1.

Hippocampal activity patterns carry information about objects in temporal context

The hippocampus is critical for human episodic memory, but its role remains controversial. One fundamental question concerns whether the hippocampus represents specific objects or assigns context-dependent representations to objects. Here, we used multivoxel pattern similarity analysis of fMRI data during retrieval of learned object sequences to systematically investigate hippocampal coding of object and temporal context information. Hippocampal activity patterns carried information about the temporal positions of objects in learned sequences, but not about objects or temporal positions in random sequences. Hippocampal activity patterns differentiated between overlapping object sequences and between temporally adjacent objects that belonged to distinct sequence contexts. Parahippocampal and perirhinal cortex showed different pattern information profiles consistent with coding of temporal position and object information, respectively. These findings are consistent with models proposing that the hippocampus represents objects within specific temporal contexts, a capability that might explain its critical role in episodic memory.

Neural activity in human hippocampal formation reveals the spatial context of retrieved memories

In many species, spatial navigation is supported by a network of place cells that exhibit increased firing whenever an animal is in a certain region of an environment. Does this neural representation of location form part of the spatiotemporal context into which episodic memories are encoded? We recorded medial temporal lobe neuronal activity as epilepsy patients performed a hybrid spatial and episodic memory task. We identified place-responsive cells active during virtual navigation and then asked whether the same cells activated during the subsequent recall of navigation-related memories without actual navigation. Place-responsive cell activity was reinstated during episodic memory retrieval. Neuronal firing during the retrieval of each memory was similar to the activity that represented the locations in the environment where the memory was initially encoded.

Simultaneous Recording of Single-Neuron Activities and Broad-Area Intracranial Electroencephalography: Electrode Design and Implantation Procedure

BACKGROUND:

There has been growing interest in clinical single-neuron recording to better understand epileptogenicity and brain function. It is crucial to compare this new information, single-neuronal activity, with that obtained from conventional intracranial electroencephalography during simultaneous recording. However, it is difficult to implant microwires and subdural electrodes during a single surgical operation because the stereotactic frame hampers flexible craniotomy.

OBJECTIVE:

To describe newly designed electrodes and surgical techniques for implanting them with subdural electrodes that enable simultaneous recording from hippocampal neurons and broad areas of the cortical surface.

METHODS:

We designed a depth electrode that does not protrude into the dura and pulsates naturally with the brain. The length and tract of the depth electrode were determined preoperatively between the lateral subiculum and the lateral surface of the temporal lobe. A frameless navigation system was used to insert the depth electrode. Surface grids and ventral strips were placed before and after the insertion of the depth electrodes, respectively. Finally, a microwire bundle was inserted into the lumen of the depth electrode. We evaluated the precision of implantation, the recording stability, and the recording rate with microwire electrodes.

RESULTS:

Depth-microwire electrodes were placed with a precision of 3.6 mm. The mean successful recording rate of single- or multiple-unit activity was 14.8%, which was maintained throughout the entire recording period.

CONCLUSION:

We achieved simultaneous implantation of microwires, depth electrodes, and broad-area subdural electrodes. Our method enabled simultaneous and stable recording of hippocampal single-neuron activities and multichannel intracranial electroencephalography.

The hippocampus, time, and memory across scales

A wealth of experimental studies with animals have offered insights about how neural networks within the hippocampus support the temporal organization of memories. These studies have revealed the existence of "time cells" that encode moments in time, much as the well-known "place cells" map locations in space. Another line of work inspired by human behavioral studies suggests that episodic memories are mediated by a state of temporal context that changes gradually over long time scales, up to at least a few thousand seconds. In this view, the "mental time travel" hypothesized to support the experience of episodic memory corresponds to a "jump back in time" in which a previous state of temporal context is recovered. We suggest that these 2 sets of findings could be different facets of a representation of temporal history that maintains a record at the last few thousand seconds of experience. The ability to represent long time scales comes at the cost of discarding precise information about when a stimulus was experienced--this uncertainty becomes greater for events further in the past. We review recent computational work that describes a mechanism that could construct such a scale-invariant representation. Taken as a whole, this suggests the hippocampus plays its role in multiple aspects of cognition by representing events embedded in a general spatiotemporal context. The representation of internal time can be useful across nonhippocampal memory systems.

Brain rhythms in mental time travel

The memory theorist Endel Tulving referred to the ability to search through one's memories, and revisit events and episodes from one's past, as mental time travel. This process involves the reactivation of past mental states reflecting the perceptual and conceptual characteristics of the original experience. Widely distributed neural circuitry is engaged in the service of memory search, and the dynamics of these circuits are reflected in rhythmic oscillatory signals at widespread frequencies, recorded both in the local field around neurons and more globally at the scalp. Retrieved-context theory provides a theoretical bridge between the behavioral phenomena exhibited by participants in memory search tasks, and the neural signals reflecting the dynamics of the underlying circuitry. Computational models based on this theory make broad predictions regarding the representational structure of neural activity recorded during these tasks. In recent work, researchers have used multivariate analytic techniques on topographic patterns of oscillatory neural activity to confirm critical predictions of retrieved-context theory. We review the cognitive theory motivating this recent work, and the analytic techniques being developed to create integrated neural-behavioral models of human memory search.