Hippocampal activity patterns carry information about objects in temporal context

Hippocampal Representations of Event Structure and Temporal Context during Episodic Temporal Order Memory

The hippocampus plays an important role in representing spatial locations and sequences and in transforming representations. How these representational structures and operations support memory for the temporal order of random items is still poorly understood. We addressed this question by leveraging the method of loci, a powerful mnemonic strategy for temporal order memory that particularly recruits hippocampus-dependent computations of spatial locations and associations. Applying representational similarity analysis to functional magnetic resonance imaging activation patterns revealed that hippocampal subfields contained representations of multiple features of sequence structure, including spatial locations, location distance, and sequence boundaries, as well as episodic-like temporal context. Critically, the hippocampal CA1 exhibited spatial transformation of representational patterns, showing lower pattern similarity for items in same locations than closely matched different locations during retrieval, whereas the CA23DG exhibited sequential transformation of representational patterns, showing lower pattern similarity for items in near locations than in far locations during encoding. These transformations enabled the encoding of multiple items in the same location and disambiguation of adjacent items. Our results suggest that the hippocampus can flexibly reconfigure multiplexed event structure representations to support accurate temporal order memory.

The Neural Representation of Ordinal Information: Domain-Specific or Domain-General?

Ordinal processing allows for the representation of the sequential relations between stimuli and is a fundamental aspect of different cognitive domains such as verbal working memory (WM), language and numerical cognition. Several studies suggest common ordinal coding mechanisms across these different domains but direct between-domain comparisons of ordinal coding are rare and have led to contradictory evidence. This fMRI study examined the commonality of ordinal representations across the WM, the number, and the letter domains by using a multivoxel pattern analysis approach and by focusing on triplet stimuli associated with robust ordinal distance effects. Neural patterns in fronto-parietal cortices distinguished ordinal distance in all domains. Critically, between-task predictions of ordinal distance in fronto-parietal cortices were robust between serial order WM, alphabetical order judgment but not when involving the numerical order judgment tasks. Moreover, frontal ROIs further supported between-task prediction of distance for the luminance judgment control task, the serial order WM, and the alphabetical tasks. These results suggest that common neural substrates characterize processing of ordinal information in WM and alphabetical but not numerical domains. This commonality, particularly in frontal cortices, may however reflect attentional control processes involved in judging ordinal distances rather than the intervention of domain-general ordinal codes.

Distinct Neural Representations of Content and Ordinal Structure in Auditory Sequence Memory

Two forms of information, frequency (content) and ordinal position (structure), have to be stored when retaining a sequence of auditory tones in working memory (WM). However, the neural representations and coding characteristics of content and structure, particularly during WM maintenance, remain elusive. Here, in two EEG studies in human participants (both sexes), by transiently perturbing the "activity-silent" WM retention state and decoding the reactivated WM information, we demonstrate that content and structure are stored in a dissociative manner with distinct characteristics throughout WM process. First, each tone in the sequence is associated with two codes in parallel, characterizing its frequency and ordinal position, respectively. Second, during retention, a structural retrocue successfully reactivates structure but not content, whereas a following white noise triggers content but not structure. Third, structure representation remains stable, whereas content code undergoes a dynamic transformation through memory progress. Finally, the noise-triggered content reactivations during retention correlate with subsequent WM behavior. Overall, our results support distinct content and structure representations in auditory WM and provide an efficient approach to access the silently stored WM information in the human brain. The dissociation of content and structure could facilitate efficient memory formation via generalizing stable structure to new auditory contents.SIGNIFICANCE STATEMENT In memory experiences, contents do not exist independently but are linked with each other via ordinal structure. For instance, recalling a piece of favorite music relies on correct ordering (sequence structure) of musical tones (content). How are the structure and content for an auditory temporally structured experience maintained in working memory? Here, by using impulse-response approach and time-resolved representational dissimilarity analysis on human EEG recordings in an auditory working memory task, we reveal that content and structure are stored in a dissociated way, which would facilitate efficient and rapid memory formation through generalizing stable structure knowledge to new auditory inputs.

Distinct cortical systems reinstate the content and context of episodic memories

Episodic recall depends upon the reinstatement of cortical activity present during the formation of a memory. Evidence from functional neuroimaging and invasive recordings in humans suggest that reinstatement organizes our memories by time or content, yet the neural systems involved in reinstating these unique types of information remain unclear. Here, combining computational modeling and intracranial recordings from 69 epilepsy patients, we show that two cortical systems uniquely reinstate the semantic content and temporal context of previously studied items during free recall. Examining either the posterior medial or anterior temporal networks, we find that forward encoding models trained on the brain's response to the temporal and semantic attributes of items can predict the serial position and semantic category of unseen items. During memory recall, these models uniquely link reinstatement of temporal context and semantic content to these posterior and anterior networks, respectively. These findings demonstrate how specialized cortical systems enable the human brain to target specific memories.

Distinct patterns of hippocampal activity associated with color and spatial source memory

The hippocampus is known to be involved in source memory across a wide variety of stimuli and source types. Thus, source memory activity in the hippocampus is thought to be domain-general such that different types of source information are similarly processed in the hippocampus. However, there is some evidence of domain-specificity for spatial and temporal source information. The current fMRI study aimed to determine whether patterns of activity in the hippocampus differed for two types of visual source information: spatial location and background color. Participants completed three runs of a spatial memory task and three runs of a color memory task. During the study phase, 32 line drawings of common objects and animals were presented to either the left or right of fixation for the spatial memory task or on either a red or green background for the color memory task. During the test phase of both tasks, 48 object word labels were presented in the center of the screen and participants classified the corresponding item as old and previously on the "left"/on a "green" background, old and previously on the "right"/on a "red" background, or "new." Two analysis methods were employed to assess whether hippocampal activity differed between the two source types: a general linear model analysis and a classification-based searchlight multivoxel pattern analysis (MVPA). The searchlight MVPA revealed that activity associated with spatial memory and color memory could be classified with above-chance accuracy in a region of the right anterior hippocampus, and a follow-up analysis revealed that there was a significant effect of memory accuracy. These results indicate that different types of source memory are represented by distinct patterns of activity in the hippocampus.

Intrinsic connectivity reveals functionally distinct cortico-hippocampal networks in the human brain

Episodic memory depends on interactions between the hippocampus and interconnected neocortical regions. Here, using data-driven analyses of resting-state functional magnetic resonance imaging (fMRI) data, we identified the networks that interact with the hippocampus-the default mode network (DMN) and a "medial temporal network" (MTN) that included regions in the medial temporal lobe (MTL) and precuneus. We observed that the MTN plays a critical role in connecting the visual network to the DMN and hippocampus. The DMN could be further divided into 3 subnetworks: a "posterior medial" (PM) subnetwork comprised of posterior cingulate and lateral parietal cortices; an "anterior temporal" (AT) subnetwork comprised of regions in the temporopolar and dorsomedial prefrontal cortex; and a "medial prefrontal" (MP) subnetwork comprised of regions primarily in the medial prefrontal cortex (mPFC). These networks vary in their functional connectivity (FC) along the hippocampal long axis and represent different kinds of information during memory-guided decision-making. Finally, a Neurosynth meta-analysis of fMRI studies suggests new hypotheses regarding the functions of the MTN and DMN subnetworks, providing a framework to guide future research on the neural architecture of episodic memory.

Sequence learning recodes cortical representations instead of strengthening initial ones

We contrast two computational models of sequence learning. The associative learner posits that learning proceeds by strengthening existing association weights. Alternatively, recoding posits that learning creates new and more efficient representations of the learned sequences. Importantly, both models propose that humans act as optimal learners but capture different statistics of the stimuli in their internal model. Furthermore, these models make dissociable predictions as to how learning changes the neural representation of sequences. We tested these predictions by using fMRI to extract neural activity patterns from the dorsal visual processing stream during a sequence recall task. We observed that only the recoding account can explain the similarity of neural activity patterns, suggesting that participants recode the learned sequences using chunks. We show that associative learning can theoretically store only very limited number of overlapping sequences, such as common in ecological working memory tasks, and hence an efficient learner should recode initial sequence representations.

Disrupted Modulation of Alpha and Low Beta Oscillations Mediates Temporal Sequence Memory Deficits in People With Schizophrenia

BACKGROUND

People with schizophrenia (SZ) exhibit impaired episodic memory when relating objects to each other in time and space. Empirical studies and computational models suggest that low-frequency neural oscillations may be a mechanism by which the brain keeps track of temporal relationships during encoding and retrieval, with modulation of oscillatory power as sequences are learned. It is unclear whether sequence memory deficits in SZ are associated with altered neural oscillations.

METHODS

Using electroencephalography, this study examined neural oscillations in 51 healthy control subjects and 37 people with SZ during a temporal sequence learning task. Multiple 5-object picture sequences were presented across 4 study-test blocks in either fixed or random order. Participants answered semantic questions for each object (e.g., living/nonliving), and sequence memory was operationalized as faster responses for fixed versus random sequences. Differences in oscillatory power between fixed versus random sequences provided a neural index of temporal sequence memory.

RESULTS

Although both groups showed reaction time differences in late blocks (blocks 3 and 4), this evidence of sequence memory was reduced in people with SZ relative to healthy control subjects. Decreases in globally distributed prestimulus alpha (8-12 Hz) and beta 1 (13-20 Hz) power for fixed versus random sequences in late blocks were also attenuated in people with SZ relative to healthy control subjects. Moreover, changes in oscillatory power predicted individual reaction time differences and fully mediated the relationship between group and sequence memory.

CONCLUSIONS

Disrupted modulation of alpha and beta 1 electroencephalography oscillations is a candidate mechanism of temporal sequence memory deficits in people with SZ.

Neural Representations of Task Context and Temporal Order During Action Sequence Execution

Routine action sequences can share a great deal of similarity in terms of their stimulus response mappings. As a consequence, their correct execution relies crucially on the ability to preserve contextual and temporal information. However, there are few empirical studies on the neural mechanism and the brain areas maintaining such information. To address this gap in the literature, we recently recorded the blood-oxygen level dependent (BOLD) response in a newly developed coffee-tea making task. The task involves the execution of four action sequences that each comprise six consecutive decision states, which allows for examining the maintenance of contextual and temporal information. Here, we report a reanalysis of this dataset using a data-driven approach, namely multivariate pattern analysis, that examines context-dependent neural activity across several predefined regions of interest. Results highlight involvement of the inferior-temporal gyrus and lateral prefrontal cortex in maintaining temporal and contextual information for the execution of hierarchically organized action sequences. Furthermore, temporal information seems to be more strongly encoded in areas over the left hemisphere.

Medial prefrontal cortex and hippocampal activity differentially contribute to ordinal and temporal context retrieval during sequence memory

Remembering sequences of events defines episodic memory, but retrieval can be driven by both ordinality and temporal contexts. Whether these modes of retrieval operate at the same time or not remains unclear. Theoretically, medial prefrontal cortex (mPFC) confers ordinality, while the hippocampus (HC) associates events in gradually changing temporal contexts. Here, we looked for evidence of each with BOLD fMRI in a sequence task that taxes both retrieval modes. To test ordinal modes, items were transferred between sequences but retained their position (e.g., AB3). Ordinal modes activated mPFC, but not HC. To test temporal contexts, we examined items that skipped ahead across lag distances (e.g., ABD). HC, but not mPFC, tracked temporal contexts. There was a mPFC and HC by retrieval mode interaction. These current results suggest that the mPFC and HC are concurrently engaged in different retrieval modes in support of remembering when an event occurred.

Temporal Regularity May Not Improve Memory for Item-Specific Detail

Regularities in event timing allow for the allocation of attention to critical time-points when an event is most likely to occur, leading to improved visual perception. Results from recent studies indicate that similar benefits may extend to memory for scenes and objects. Here, we investigated whether benefits of temporal regularity are evident when detailed, item-specific representations are necessary for successful recognition memory performance. In Experiments 1 and 2, pictures of objects were presented with either predictable or randomized event timing, in separate encoding blocks. In the test phase, old and new objects were presented, intermixed with perceptually similar exemplars of encoded objects. In Experiment 3 we attempted to replicate previously reported memory enhancements for scenes. In contrast to predictions, temporal regularity did not affect response times (RT) or improve recognition memory accuracy in any of our experiments. These results suggest that any effects of temporal expectation on memory are subtle and may be sensitive to minor changes in task parameters. In sum, indirect upregulation of attention through imposed temporal structure may not be sufficient to have downstream effects on memory performance.

Linking the Rapid Cascade of Visuo-Attentional Processes to Successful Memory Encoding

While it is broadly accepted that attention modulates memory, the contribution of specific rapid attentional processes to successful encoding is largely unknown. To investigate this issue, we leveraged the high temporal resolution of electroencephalographic recordings to directly link a cascade of visuo-attentional neural processes to successful encoding: namely (1) the N2pc (peaking ~200 ms), which reflects stimulus-specific attentional orienting and allocation, (2) the sustained posterior-contralateral negativity (post-N2pc), which has been associated with sustained visual processing, (3) the contralateral reduction in oscillatory alpha power (contralateral reduction in alpha > 200 ms), which has also been independently related to attentionally sustained visual processing. Each of these visuo-attentional processes was robustly predictive of successful encoding, and, moreover, each enhanced memory independently of the classic, longer-latency, conceptually related, difference-due-to memory (Dm) effect. Early latency midfrontal theta power also promoted successful encoding, with at least part of this influence being mediated by the later latency Dm effect. These findings markedly expand current knowledge by helping to elucidate the intimate relationship between attentional modulations of perceptual processing and effective encoding for later memory retrieval.

The brain in motion: How ensemble fluidity drives memory-updating and flexibility

While memories are often thought of as flashbacks to a previous experience, they do not simply conserve veridical representations of the past but must continually integrate new information to ensure survival in dynamic environments. Therefore, 'drift' in neural firing patterns, typically construed as disruptive 'instability' or an undesirable consequence of noise, may actually be useful for updating memories. In our view, continual modifications in memory representations reconcile classical theories of stable memory traces with neural drift. Here we review how memory representations are updated through dynamic recruitment of neuronal ensembles on the basis of excitability and functional connectivity at the time of learning. Overall, we emphasize the importance of considering memories not as static entities, but instead as flexible network states that reactivate and evolve across time and experience.

The impact of encoding scenarios on different forms of temporal order memory

This study investigated the impact of encoding scenarios on sequential memory, specifcally testing the benefit of imagining a spatial context when learning a list of items. In Experiment 1, participants studied sequences of visually presented items after imagining a spatial context or a preparatory cue. At test, participants made recency judgments (Which came first?) on pairs of items from the studied sequences. These test pairs varied in the studied temporal distance (studied with 0, 1 or 2 intervening items between the pair) and whether presented in a manner that was 'spatially congruent' with their studied order. A congruent test pair was one in which the first item from the sequence was presented on the left and the second item presented on the right. The results were that imagining a spatial context at encoding led to more accurate recency judgments than when no encoding context was provided. In addition, test pairs that included items studied farther apart (2 intervening items) were more accurately judged than test pairs with items studied closer together (0 or 1 intervening items). Experiment 2 utilized the same design to compare the effect of imagining a spatial context versus elaborating on a concept when encoding item sequences. Interestingly, the effect of these encoding contexts on recency judgments depended on both the temporal distance and congruency of the test pairs. Imagining a spatial context at encoding led to more accurate recency judgments than when thinking about a concept at encoding for incongruent test pairs and congruent test pairs studied without intervening items. These findings provide new insight into how encoding strategies, specifically those that recruit imagining spatial contexts, alter the way temporal associations are retrieved from memory.

Differential Representations of Perceived and Retrieved Visual Information in Hippocampus and Cortex

Memory retrieval is thought to depend on interactions between hippocampus and cortex, but the nature of representation in these regions and their relationship remains unclear. Here, we performed an ultra-high field fMRI (7T) experiment, comprising perception, learning and retrieval sessions. We observed a fundamental difference between representations in hippocampus and high-level visual cortex during perception and retrieval. First, while object-selective posterior fusiform cortex showed consistent responses that allowed us to decode object identity across both perception and retrieval one day after learning, object decoding in hippocampus was much stronger during retrieval than perception. Second, in visual cortex but not hippocampus, there was consistency in response patterns between perception and retrieval, suggesting that substantial neural populations are shared for both perception and retrieval. Finally, the decoding in hippocampus during retrieval was not observed when retrieval was tested on the same day as learning suggesting that the retrieval process itself is not sufficient to elicit decodable object representations. Collectively, these findings suggest that while cortical representations are stable between perception and retrieval, hippocampal representations are much stronger during retrieval, implying some form of reorganization of the representations between perception and retrieval.

Time as the fourth dimension in the hippocampus

Experiences of animal and human beings are structured by the continuity of space and time coupled with the uni-directionality of time. In addition to its pivotal position in spatial processing and navigation, the hippocampal system also plays a central, multiform role in several types of temporal processing. These include timing and sequence learning, at scales ranging from meso-scales of seconds to macro-scales of minutes, hours, days and beyond, encompassing the classical functions of short term memory, working memory, long term memory, and episodic memories (comprised of information about when, what, and where). This review article highlights the principal findings and behavioral contexts of experiments in rats showing: 1) timing: tracking time during delays by hippocampal 'time cells' and during free behavior by hippocampal-afferent lateral entorhinal cortex ramping cells; 2) 'online' sequence processing: activity coding sequences of events during active behavior; 3) 'off-line' sequence replay: during quiescence or sleep, orderly reactivation of neuronal assemblies coding awake sequences. Studies in humans show neurophysiological correlates of episodic memory comparable to awake replay. Neural mechanisms are discussed, including ion channel properties, plateau and ramping potentials, oscillations of excitation and inhibition of population activity, bursts of high amplitude discharges (sharp wave ripples), as well as short and long term synaptic modifications among and within cell assemblies. Specifically conceived neural network models will suggest processes supporting the emergence of scalar properties (Weber's law), and include different classes of feedforward and recurrent network models, with intrinsic hippocampal coding for 'transitions' (sequencing of events or places).

Temporal context guides visual exploration during scene recognition

Memories for episodes are temporally structured. Cognitive models derived from list-learning experiments attribute this structure to the retrieval of temporal context information that indicates when a memory occurred. These models predict key features of memory recall, such as the strong tendency to retrieve studied items in the order in which they were first encountered. Can such models explain ecological memory behaviors, such as eye movements during encoding and retrieval of complex visual stimuli? We tested predictions from retrieved-context models using three data sets involving recognition memory and free viewing of complex scenes. Subjects reinstated sequences of eye movements from one scene-viewing episode to the next. Moreover, sequence reinstatement decayed over time and was associated with successful memory. We observed memory-driven reinstatement even after accounting for intrinsic scene properties that produced consistent eye movements. These findings confirm predictions of retrieved-context models, suggesting retrieval of temporal context influences complex behaviors generated during naturalistic memory experiences. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Hippocampal-striatal functional connectivity supports processing of temporal expectations from associative memory

The hippocampus and dorsal striatum are both associated with temporal processing, but they are thought to play distinct roles. The hippocampus has been reported to contribute to storing temporal structure of events in memory, whereas the striatum contributes to temporal motor preparation and reward anticipation. Here, we asked whether the striatum cooperates with the hippocampus in processing the temporal context of memorized visual associations. In our task, participants were trained to implicitly form temporal expectations for one of two possible time intervals associated to specific cue-target associations, and subsequently were scanned using ultra-high-field 7T functional magnetic resonance imaging. During scanning, learned temporal expectations could be violated when the pairs were presented at either the associated or not-associated time intervals. When temporal expectations were met during testing trials, activity in left and right hippocampal subfields and right putamen decreased, compared to when temporal expectations were not met. Further, psycho-physiological interactions showed that functional connectivity between left hippocampal subfields and caudate decreased when temporal expectations were not met. Our results indicate that the hippocampus and striatum cooperate to process implicit temporal expectation from mnemonic associations. Our findings provide further support for a hippocampal-striatal network in temporal associative processing.

Hierarchical Representation of Multistep Tasks in Multiple-Demand and Default Mode Networks

Task episodes consist of sequences of steps that are performed to achieve a goal. We used fMRI to examine neural representation of task identity, component items, and sequential position, focusing on two major cortical systems—the multiple-demand (MD) and default mode networks (DMN). Human participants (20 males, 22 females) learned six tasks each consisting of four steps. Inside the scanner, participants were cued which task to perform and then sequentially identified the target item of each step in the correct order. Univariate time course analyses indicated that intra-episode progress was tracked by a tonically increasing global response, plus an increasing phasic step response specific to MD regions. Inter-episode boundaries evoked a widespread response at episode onset, plus a marked offset response specific to DMN regions. Representational similarity analysis (RSA) was used to examine representation of task identity and component steps. Both networks represented the content and position of individual steps, however the DMN preferentially represented task identity while the MD network preferentially represented step-level information. Thus, although both MD and DMN networks are sensitive to step-level and episode-level information in the context of hierarchical task performance, they exhibit dissociable profiles in terms of both temporal dynamics and representational content. The results suggest collaboration of multiple brain regions in control of multistep behavior, with MD regions particularly involved in processing the detail of individual steps, and DMN adding representation of broad task context.

SIGNIFICANCE STATEMENT Achieving one's goals requires knowing what to do and when. Tasks are typically hierarchical, with smaller steps nested within overarching goals. For effective, flexible behavior, the brain must represent both levels. We contrast response time courses and information content of two major cortical systems—the multiple-demand (MD) and default mode networks (DMN)—during multistep task episodes. Both networks are sensitive to step-level and episode-level information, but with dissociable profiles. Intra-episode progress is tracked by tonically increasing global responses, plus MD-specific increasing phasic step responses. Inter-episode boundaries evoke widespread responses at episode onset, plus DMN-specific offset responses. Both networks represent content and position of individual steps; however, the DMN and MD networks favor task identity and step-level information, respectively.

The developmental neural substrates of Hebb repetition learning and their link with reading ability

Hebb repetition learning is a fundamental learning mechanism for sequential knowledge, such as language. However, still little is known about its development. This fMRI study examined the developmental neural substrates of Hebb repetition learning and its relation with reading abilities in a group of 49 children aged from 6 to 12 years. In the scanner, the children carried out an immediate serial recall task for syllable sequences of which some sequences were repeated several times over the course of the session (Hebb repetition sequences). The rate of Hebb repetition learning was associated with modulation of activity in the medial temporal lobe. Importantly, for the age range studied here, learning-related medial temporal lobe modulation was independent of the age of the children. Furthermore, we observed an association between regular and irregular word reading abilities and the neural substrates of Hebb repetition learning. This study suggests that the functional neural substrates of Hebb repetition learning do not undergo further maturational changes in school age children, possibly because they are sustained by implicit sequential learning mechanisms which are considered to be fully developed by that age. Importantly, the neural substrates of Hebb learning remain significant determinants of children's learning abilities, such as reading.

Sequence Memory in the Hippocampal-Entorhinal Region

Episodic memories are constructed from sequences of events. When recalling such a memory, we not only recall individual events, but we also retrieve information about how the sequence of events unfolded. Here, we focus on the role of the hippocampal-entorhinal region in processing and remembering sequences of events, which are thought to be stored in relational networks. We summarize evidence that temporal relations are a central organizational principle for memories in the hippocampus. Importantly, we incorporate novel insights from recent studies about the role of the adjacent entorhinal cortex in sequence memory. In rodents, the lateral entorhinal subregion carries temporal information during ongoing behavior. The human homologue is recruited during memory recall where its representations reflect the temporal relationships between events encountered in a sequence. We further introduce the idea that the hippocampal-entorhinal region might enable temporal scaling of sequence representations. Flexible changes of sequence progression speed could underlie the traversal of episodic memories and mental simulations at different paces. In conclusion, we describe how the entorhinal cortex and hippocampus contribute to remembering event sequences-a core component of episodic memory.

Three brain states in the hippocampus and cortex

Contemporary brain research seeks to understand how cognition is reducible to neural activity. Crucially, much of this effort is guided by a scientific paradigm that views neural activity as essentially driven by external stimuli. In contrast, recent perspectives argue that this paradigm is by itself inadequate and that understanding patterns of activity intrinsic to the brain is needed to explain cognition. Yet, despite this critique, the stimulus-driven paradigm still dominates-possibly because a convincing alternative has not been clear. Here, we review a series of findings suggesting such an alternative. These findings indicate that neural activity in the hippocampus occurs in one of three brain states that have radically different anatomical, physiological, representational, and behavioral correlates, together implying different functional roles in cognition. This three-state framework also indicates that neural representations in the hippocampus follow a surprising pattern of organization at the timescale of ∼1 s or longer. Lastly, beyond the hippocampus, recent breakthroughs indicate three parallel states in the cortex, suggesting shared principles and brain-wide organization of intrinsic neural activity.

Multi-level analyses of associative recognition memory: the whole is greater than the sum of its parts

Associative recognition memory depends on the integration of information concerning an item and the spatio-temporal context in which it was encountered. Such an integration depends on dynamic interactions across a brain-wide memory network. Here we discuss evidence from multiple levels of analysis, behavioural, cellular and synaptic which demonstrating the existence of multiple overlapping, subnetworks embedded within these large-scale networks. Recent advances have revealed that of these subnetworks, a distinct hippocampal-prefrontal networks are engaged by different representations (object-spatial or object temporal). Other subnetworks are recruited by distinct processing demands, such as encoding and retrieval which are supported by distinct cellular and synaptic processes. One challenge to multi-level investigations of memory continues to be that conclusions are drawn from correlations of effects rather than from direct evidence of causation.

The Human Medial Temporal Lobe Is Necessary for Remembering Durations within a Sequence of Events but Not Durations of Individual Events

Recent interest in the role of the hippocampus in temporal aspects of cognition has been fueled, in part, by the observation of “time” cells in the rodent hippocampus—that is, cells that have differential firing patterns depending on how long ago an event occurred. Such cells are thought to provide an internal representation of elapsed time. Yet, the hippocampus is not needed for processing temporal duration information per se, at least on the order of seconds, as evidenced by intact duration judgments in rodents and humans with hippocampal damage. Rather, it has been proposed that the hippocampus may be essential for coding higher order aspects of temporal mnemonic processing, such as those needed to temporally organize a sequence of events that form an episode. To examine whether (1) the hippocampus uses duration information in the service of establishing temporal relations among events and (2) its role in memory for duration is unique to sequences, we tested amnesic patients with medial-temporal lobe damage (including the hippocampus). We hypothesized that medial-temporal lobe damage should impair the ability to remember sequential duration information but leave intact judgments about duration devoid of a sequential demand. We found that amnesics were impaired in making judgments about durations within a sequence but not in judging single durations. This impairment was not due to higher cognitive load associated with duration judgments about sequences. In convergence with rodent and human fMRI work, these findings shed light on how time coding in the hippocampus may contribute to temporal cognition.

The hippocampus, prefrontal cortex, and perirhinal cortex are critical to incidental order memory

Considerable research in rodents and humans indicates the hippocampus and prefrontal cortex are essential for remembering temporal relationships among stimuli, and accumulating evidence suggests the perirhinal cortex may also be involved. However, experimental parameters differ substantially across studies, which limits our ability to fully understand the fundamental contributions of these structures. In fact, previous studies vary in the type of temporal memory they emphasize (e.g., order, sequence, or separation in time), the stimuli and responses they use (e.g., trial-unique or repeated sequences, and incidental or rewarded behavior), and the degree to which they control for potential confounding factors (e.g., primary and recency effects, or order memory deficits secondary to item memory impairments). To help integrate these findings, we developed a new paradigm testing incidental memory for trial-unique series of events, and concurrently assessed order and item memory in animals with damage to the hippocampus, prefrontal cortex, or perirhinal cortex. We found that this new approach led to robust order and item memory, and that hippocampal, prefrontal and perirhinal damage selectively impaired order memory. These findings suggest the hippocampus, prefrontal cortex and perirhinal cortex are part of a broad network of structures essential for incidentally learning the order of events in episodic memory.

Developmental differences in temporal schema acquisition impact reasoning decisions

Schemas capture patterns across multiple experiences, accumulating information about common event structures that guide decision making in new contexts. Schemas are an important principle of leading theories of cognitive development; yet, we know little about how children and adolescents form schemas and use schematic knowledge to guide decisions. Here, we show that the ability to acquire schematic knowledge based on the temporal regularities of events increases during childhood and adolescence. Furthermore, we show that temporally mediated schematic knowledge biases reasoning decisions in an age-dependent manner. Participants with greater temporal schematic knowledge were more likely to infer that temporally related items shared other, non-temporal properties, with adults showing the greatest relationship between schema knowledge and reasoning choices. These data indicate that the mechanisms underlying schema formation and expression are not fully developed until adulthood and may reflect the ongoing maturation of hippocampus and prefrontal cortex through adolescence.

Event Perception and Memory

Events make up much of our lived experience, and the perceptual mechanisms that represent events in experience have pervasive effects on action control, language use, and remembering. Event representations in both perception and memory have rich internal structure and connections one to another, and both are heavily informed by knowledge accumulated from previous experiences. Event perception and memory have been identified with specific computational and neural mechanisms, which show protracted development in childhood and are affected by language use, expertise, and brain disorders and injuries. Current theoretical approaches focus on the mechanisms by which events are segmented from ongoing experience, and emphasize the common coding of events for perception, action, and memory. Abetted by developments in eye-tracking, neuroimaging, and computer science, research on event perception and memory is moving from small-scale laboratory analogs to the complexity of events in the wild.

The hippocampus contributes to temporal duration memory in the context of event sequences: A cross-species perspective

Although a large body of research has implicated the hippocampus in the processing of memory for temporal duration, there is an exigent degree of inconsistency across studies that obfuscates the precise contributions of this structure. To shed light on this issue, the present review article surveys both historical and recent cross-species evidence emanating from a wide variety of experimental paradigms, identifying areas of convergence and divergence. We suggest that while factors such as time-scale (e.g. the length of durations involved) and the nature of memory processing (e.g. prospective vs. retrospective memory) are very helpful in the interpretation of existing data, an additional important consideration is the context in which the duration information is experienced and processed, with the hippocampus being preferentially involved in memory for durations that are embedded within a sequence of events. We consider the mechanisms that may underpin temporal duration memory and how the same mechanisms may contribute to memory for other aspects of event sequences such as temporal order.

The hippocampus as a visual area organized by space and time: A spatiotemporal similarity hypothesis

The hippocampus is the canonical memory system in the brain and is not typically considered part of the visual system. Yet, it sits atop the ventral visual stream and has been implicated in certain aspects of vision. Here I review the place of the hippocampal memory system in vision science. After a brief primer on the local circuity, external connectivity, and computational functions of the hippocampus, I explore what can be learned from each field about the other. I first present four areas of vision science (scene perception, imagery, eye movements, attention) that challenge our current understanding of the hippocampus in terms of its role in episodic memory. In the reverse direction, I leverage this understanding to inform vision science in other ways, presenting a working hypothesis about a unique form of visual representation. This spatiotemporal similarity hypothesis states that the hippocampus represents objects according to whether they co-occur in space and/or time, and not whether they look alike, as elsewhere in the visual system. This tuning may reflect hippocampal mechanisms of pattern separation, relational binding, and statistical learning, allowing the hippocampus to generate visual expectations to facilitate search and recognition.

Focusing on what matters: Modulation of the human hippocampus by relational attention

Hippocampal episodic memory is fundamentally relational, comprising links between events and the spatiotemporal contexts in which they occurred. Such relations are also important over shorter timescales, during online perception. For example, how do we assess the relative spatial positions of objects, their temporal order, or the relationship between their features? Here, we investigate the role of the hippocampus in online relational processing by manipulating attention to different kinds of relations. While undergoing fMRI, participants viewed two images in rapid succession on each trial and performed one of three relational tasks, judging the images' relative: spatial positions, temporal onsets, or sizes. Additionally, they sometimes judged whether one image was tilted, irrespective of the other. This served as a baseline item task with no demands on relational processing. The hippocampus showed reliable deactivation when participants attended to relational vs. item information. Attention to temporal relations was associated with the most robust deactivation. One interpretation of such deactivation is that it reflects hippocampal disengagement. If true, there should be reduced information content and noisier activity patterns for the temporal vs. other tasks. Instead, multivariate pattern analysis revealed more stable hippocampal representations in the temporal task. This increased pattern similarity was not simply a reflection of lower univariate activity. Thus, the hippocampus differentiates between relational and item processing even during online perception, and its representations of temporal relations are particularly robust. These findings suggest that the relational computations of the hippocampus extend beyond long-term memory, enabling rapid extraction of relational information in perception.

Complementary Task Structure Representations in Hippocampus and Orbitofrontal Cortex during an Odor Sequence Task

Both hippocampus (HPC) and orbitofrontal cortex (OFC) have been shown to be critical for behavioral tasks that require use of an internal model or cognitive map, composed of the states and the relationships between them, which define the current environment or task at hand. One general idea is that the HPC provides the cognitive map, which is then transformed by OFC to emphasize information of relevance to current goals. Our previous analysis of ensemble activity in OFC in rats performing an odor sequence task revealed a rich representation of behaviorally relevant task structure, consistent with this proposal. Here, we compared those data to recordings from single units in area CA1 of the HPC of rats performing the same task. Contrary to expectations that HPC ensembles would represent detailed, even incidental, information defining the full task space, we found that HPC ensembles-like those in OFC-failed to distinguish states when it was not behaviorally necessary. However, hippocampal ensembles were better than those in OFC at distinguishing task states in which prospective memory was necessary for future performance. These results suggest that, in familiar environments, the HPC and OFC may play complementary roles, with the OFC maintaining the subjects' current position on the cognitive map or state space, supported by HPC when memory demands are high.

Learning predictive structure without a teacher: decision strategies and brain routes

Extracting the structure of complex environments is at the core of our ability to interpret the present and predict the future. This skill is important for a range of behaviours from navigating a new city to learning music and language. Classical approaches that investigate our ability to extract the principles of organisation that govern complex environments focus on reward-based learning. Yet, the human brain is shown to be expert at learning generative structure based on mere exposure and without explicit reward. Individuals are shown to adapt to-unbeknownst to them-changes in the environment's temporal statistics and predict future events. Further, we present evidence for a common brain architecture for unsupervised structure learning and reward-based learning, suggesting that the brain is built on the premise that 'learning is its own reward' to support adaptive behaviour.

Hippocampal connectivity with sensorimotor cortex during volitional finger movements: Laterality and relationship to motor learning

Hippocampal interactions with the motor system are often assumed to reflect the role of memory in motor learning. Here, we examine hippocampal connectivity with sensorimotor cortex during two tasks requiring paced movements, one with a mnemonic component (sequence learning) and one without (repetitive tapping). Functional magnetic resonance imaging activity was recorded from thirteen right-handed subjects; connectivity was identified from sensorimotor cortex correlations with psychophysiological interactions in hippocampal activity between motor and passive visual tasks. Finger movements in both motor tasks anticipated the timing of the metronome, reflecting cognitive control, yet evidence of motor learning was limited to the sequence learning task; nonetheless, hippocampal connectivity was observed during both tasks. Connectivity from corresponding regions in the left and right hippocampus overlapped extensively, with improved sensitivity resulting from their conjunctive (global) analysis. Positive and negative connectivity were both evident, with positive connectivity in sensorimotor cortex ipsilateral to the moving hand during unilateral movements, whereas negative connectivity was prominent in whichever hemisphere was most active during movements. Results implicate the hippocampus in volitional finger movements even in the absence of motor learning or recall.

Temporal features of spatial knowledge: Representing order and duration of topographical information

Environmental navigation entails the constant integration of information across space and time; however, the relation between spatial and temporal features involved in wayfinding has not been fully established yet. Here we investigated how two key spatio-temporal aspects of navigation - namely the processing of information concerning the order of landmarks along a route, and the duration of tracts connecting the same landmarks - relate to different types of navigational learning. Participants encoded a path in a real city in both a route and a survey format, and the acquisition of landmark, route and survey knowledge was tested. Participants' knowledge of landmarks order, and their perception of tracts duration were also assessed. Performance in the survey task, but not in the landmark and route tasks, significantly predicted accuracy in landmark ordering. The influence of tract length on retrospectively estimated tracts duration was also found to be significantly predicted only by accuracy in the survey learning task. These results support recent models of spatial navigation, invoking the dynamic interaction between different representation formats. Furthermore, they are consistent with theoretical views of an integrated account of the role of the hippocampus in navigation and memory.

In a Temporally Segmented Experience Hippocampal Neurons Represent Temporally Drifting Context But Not Discrete Segments

There is widespread agreement that episodic memory is organized into a timeline of past experiences. Recent work suggests that the hippocampus may parse the flow of experience into discrete episodes separated by event boundaries. A complementary body of work suggests that context changes gradually as experience unfolds. We recorded from hippocampal neurons as male Long-Evans rats performed 6 blocks of an object discrimination task in sets of 15 trials. Each block was separated by removal from the testing chamber for a delay to enable segmentation. The reward contingency reversed from one block to the next to incentivize segmentation. We expected animals to hold two distinct, recurring representations of context to match the two distinct rule contingencies. Instead, we found that overtrained rats began each block neither above nor below chance but by guessing randomly. While many units had clear firing fields selective to the conjunction of objects in places, a significant population also reflected a continuously drifting code both within block and across blocks. Despite clear boundaries between blocks, we saw no neural evidence for event segmentation in this experiment. Rather, the hippocampal ensemble drifted continuously across time. This continuous drift in the neural representation was consistent with the lack of segmentation observed in behavior.SIGNIFICANCE STATEMENT The neuroscience literature yet to reach consensus on how the hippocampus supports the organization of events across time in episodic memory. Initial studies reported stable hippocampal maps segmented by remapping events. However, it remains unclear whether segmentation is an artifact of cue responsivity. Recently, research has shown that the hippocampal code exhibits continuous drift. Drift may represent a continually evolving context; however, it is unclear whether this is an artifact of changing experiences. We recorded dCA1 in rats performing an object discrimination task designed to segment time. Overtrained rats could not anticipate upcoming context switches but used context boundaries to their advantage. Hippocampal ensembles showed neither evidence of alternating between stable contexts nor sensitivity to boundaries, but showed robust temporal drift.

Associative binding in early childhood: Evidence from a preferential looking paradigm

Relational memory processes are responsible for forming representations that bind together the co-occurring elements of an experience. These processes provide a foundation of episodic memory, the capacity to remember specific events about one's past. In the present research, we used a visual comparison paradigm to determine whether toddlers can form memories for the relation between co-occurring items. In Experiment 1, 66 2-year-olds viewed pairs of cartoon faces (AB; CD) and did not show a significant novelty preference for the violated associations (i.e., did not look longer at AD rearranged pair than old AB pair when exposed to AD-AB). However, toddlers looked longest at the individual face (i.e., D) violating a studied pair, which appeared to be supported by visual preference for centrally located stimuli. In Experiment 2, 46 2-year-olds participated in a similar procedure, but faces violating the pair during test were located to the periphery (e.g., AB-AC). Under these conditions, toddlers looked longer at recombined pair AC. Overall, our results show that toddlers show some ability to make item-item associations but may need to overcome preferential looking biases to demonstrate this capacity. Furthermore, looking behaviors beyond overall novelty preference may be informative even when overall novelty preference is not found.

Mapping sequence structure in the human lateral entorhinal cortex

Remembering event sequences is central to episodic memory and presumably supported by the hippocampal-entorhinal region. We previously demonstrated that the hippocampus maps spatial and temporal distances between events encountered along a route through a virtual city (Deuker et al., 2016), but the content of entorhinal mnemonic representations remains unclear. Here, we demonstrate that multi-voxel representations in the anterior-lateral entorhinal cortex (alEC) - the human homologue of the rodent lateral entorhinal cortex - specifically reflect the temporal event structure after learning. Holistic representations of the sequence structure related to memory recall and the timeline of events could be reconstructed from entorhinal multi-voxel patterns. Our findings demonstrate representations of temporal structure in the alEC; dovetailing with temporal information carried by population signals in the lateral entorhinal cortex of navigating rodents and alEC activations during temporal memory retrieval. Our results provide novel evidence for the role of the alEC in representing time for episodic memory.

Human Replay Spontaneously Reorganizes Experience

Knowledge abstracted from previous experiences can be transferred to aid new learning. Here, we asked whether such abstract knowledge immediately guides the replay of new experiences. We first trained participants on a rule defining an ordering of objects and then presented a novel set of objects in a scrambled order. Across two studies, we observed that representations of these novel objects were reactivated during a subsequent rest. As in rodents, human "replay" events occurred in sequences accelerated in time, compared to actual experience, and reversed their direction after a reward. Notably, replay did not simply recapitulate visual experience, but followed instead a sequence implied by learned abstract knowledge. Furthermore, each replay contained more than sensory representations of the relevant objects. A sensory code of object representations was preceded 50 ms by a code factorized into sequence position and sequence identity. We argue that this factorized representation facilitates the generalization of a previously learned structure to new objects.

Episodic memory: Neuronal codes for what, where, and when

Episodic memory is defined as the ability to recall events in a spatiotemporal context. Formation of such memories is critically dependent on the hippocampal formation and its inputs from the entorhinal cortex. To be able to support the formation of episodic memories, entorhinal cortex and hippocampal formation should contain a neuronal code that follows several requirements. First, the code should include information about position of the agent ("where"), sequence of events ("when"), and the content of the experience itself ("what"). Second, the code should arise instantly thereby being able to support memory formation of one-shot experiences. For successful encoding and to avoid interference between memories during recall, variations in location, time, or in content of experience should result in unique ensemble activity. Finally, the code should capture several different resolutions of experience so that the necessary details relevant for future memory-based predictions will be stored. We review how neuronal codes in entorhinal cortex and hippocampus follow these requirements and argue that during formation of episodic memories entorhinal cortex provides hippocampus with instant information about ongoing experience. Such information originates from (a) spatially modulated neurons in medial entorhinal cortex, including grid cells, which provide a stable and universal positional metric of the environment; (b) a continuously varying signal in lateral entorhinal cortex providing a code for the temporal progression of events; and (c) entorhinal neurons coding the content of experiences exemplified by object-coding and odor-selective neurons. During formation of episodic memories, information from these systems are thought to be encoded as unique sequential ensemble activity in hippocampus, thereby encoding associations between the content of an event and its spatial and temporal contexts. Upon exposure to parts of the encoded stimuli, activity in these ensembles can be reinstated, leading to reactivation of the encoded activity pattern and memory recollection.

Prefrontal Pathways Provide Top-Down Control of Memory for Sequences of Events

We remember our lives as sequences of events, but it is unclear how these memories are controlled during retrieval. In rats, the medial prefrontal cortex (mPFC) is positioned to influence sequence memory through extensive top-down inputs to regions heavily interconnected with the hippocampus, notably the nucleus reuniens of the thalamus (RE) and perirhinal cortex (PER). Here, we used an hM4Di synaptic-silencing approach to test our hypothesis that specific mPFC→RE and mPFC→PER projections regulate sequence memory retrieval. First, we found non-overlapping populations of mPFC cells project to RE and PER. Second, suppressing mPFC activity impaired sequence memory. Third, inhibiting mPFC→RE and mPFC→PER pathways effectively abolished sequence memory. Finally, a sequential lag analysis showed that the mPFC→RE pathway contributes to a working memory retrieval strategy, whereas the mPFC→PER pathway supports a temporal context memory retrieval strategy. These findings demonstrate that mPFC→RE and mPFC→PER pathways serve as top-down mechanisms that control distinct sequence memory retrieval strategies.

Schema and Motor-Memory Consolidation

Recent research has demonstrated that memory-consolidation processes can be accelerated if newly learned information is consistent with preexisting knowledge. Until now, investigations of this fast integration of new information into memory have focused on the declarative and perceptual systems. We employed a unique manipulation of a motor-sequence-learning paradigm to examine the effect of experimentally acquired memory on the learning of new motor information. Results demonstrate that new information is rapidly integrated into memory when practice occurs in a framework that is compatible with the previously acquired memory. This framework consists of the ordinal representation of the motor sequence. This enhanced integration cannot be explained by differences in the explicit awareness of the sequence and is observed only if the previously acquired motor memory was consolidated overnight. Results are consistent with the schema model of memory consolidation and offer insights into how previous motor experience can accelerate learning and consolidation processes.

Navigating cognition: Spatial codes for human thinking

The hippocampal formation has long been suggested to underlie both memory formation and spatial navigation. We discuss how neural mechanisms identified in spatial navigation research operate across information domains to support a wide spectrum of cognitive functions. In our framework, place and grid cell population codes provide a representational format to map variable dimensions of cognitive spaces. This highly dynamic mapping system enables rapid reorganization of codes through remapping between orthogonal representations across behavioral contexts, yielding a multitude of stable cognitive spaces at different resolutions and hierarchical levels. Action sequences result in trajectories through cognitive space, which can be simulated via sequential coding in the hippocampus. In this way, the spatial representational format of the hippocampal formation has the capacity to support flexible cognition and behavior.

Medial Orbitofrontal Cortex, Dorsolateral Prefrontal Cortex, and Hippocampus Differentially Represent the Event Saliency

Two primary functions attributed to the hippocampus and prefrontal cortex (PFC) network are retaining the temporal and spatial associations of events and detecting deviant events. It is unclear, however, how these two functions converge into one mechanism. Here, we tested whether increased activity with perceiving salient events is a deviant detection signal or contains information about the event associations by reflecting the magnitude of deviance (i.e., event saliency). We also tested how the deviant detection signal is affected by the degree of anticipation. We studied regional neural activity when people watched a movie that had varying saliency of a novel or an anticipated flow of salient events. Using intracranial electroencephalography from 10 patients, we observed that high-frequency activity (50-150 Hz) in the hippocampus, dorsolateral PFC, and medial OFC tracked event saliency. We also observed that medial OFC activity was stronger when the salient events were anticipated than when they were novel. These results suggest that dorsolateral PFC and medial OFC, as well as the hippocampus, signify the saliency magnitude of events, reflecting the hierarchical structure of event associations.

Evidence for the incorporation of temporal duration information in human hippocampal long-term memory sequence representations

There has been much interest in how the hippocampus codes time in support of episodic memory. Notably, while rodent hippocampal neurons, including populations in subfield CA1, have been shown to represent the passage of time in the order of seconds between events, there is limited support for a similar mechanism in humans. Specifically, there is no clear evidence that human hippocampal activity during long-term memory processing is sensitive to temporal duration information that spans seconds. To address this gap, we asked participants to first learn short event sequences that varied in image content and interval durations. During fMRI, participants then completed a recognition memory task, as well as a recall phase in which they were required to mentally replay each sequence in as much detail as possible. We found that individual sequences could be classified using activity patterns in the anterior hippocampus during recognition memory. Critically, successful classification was dependent on the conjunction of event content and temporal structure information (with unsuccessful classification of image content or interval duration alone), and further analyses suggested that the most informative voxels resided in the anterior CA1. Additionally, a classifier trained on anterior CA1 recognition data could successfully identify individual sequences from the mental replay data, suggesting that similar activity patterns supported participants' recognition and recall memory. Our findings complement recent rodent hippocampal research, and provide evidence that long-term sequence memory representations in the human hippocampus can reflect duration information in the order of seconds.