Reward revaluation biases hippocampal replay content away from the preferred outcome

Neurofeedback training can modulate task-relevant memory replay rate in rats

Hippocampal replay - the time-compressed, sequential reactivation of ensembles of neurons related to past experience - is a key neural mechanism of memory consolidation. Replay typically coincides with a characteristic pattern of local field potential activity, the sharp-wave ripple (SWR). Reduced SWR rates are associated with cognitive impairment in multiple models of neurodegenerative disease, suggesting that a clinically viable intervention to promote SWRs and replay would prove beneficial. We therefore developed a neurofeedback paradigm for rat subjects in which SWR detection triggered rapid positive feedback in the context of a memory-dependent task. This training protocol increased the prevalence of task-relevant replay during the targeted neurofeedback period by changing the temporal dynamics of SWR occurrence. This increase was also associated with neural and behavioral forms of compensation after the targeted period. These findings reveal short-timescale regulation of SWR generation and demonstrate that neurofeedback is an effective strategy for modulating hippocampal replay.

A recurrent network model of planning explains hippocampal replay and human behavior

When faced with a novel situation, people often spend substantial periods of time contemplating possible futures. For such planning to be rational, the benefits to behavior must compensate for the time spent thinking. Here, we capture these features of behavior by developing a neural network model where planning itself is controlled by the prefrontal cortex. This model consists of a meta-reinforcement learning agent augmented with the ability to plan by sampling imagined action sequences from its own policy, which we call 'rollouts'. In a spatial navigation task, the agent learns to plan when it is beneficial, which provides a normative explanation for empirical variability in human thinking times. Additionally, the patterns of policy rollouts used by the artificial agent closely resemble patterns of rodent hippocampal replays. Our work provides a theory of how the brain could implement planning through prefrontal-hippocampal interactions, where hippocampal replays are triggered by-and adaptively affect-prefrontal dynamics.

Paradoxical replay can protect contextual task representations from destructive interference when experience is unbalanced

Experience replay is a powerful mechanism to learn efficiently from limited experience. Despite several decades of compelling experimental results, the factors that determine which experiences are selected for replay remain unclear. A particular challenge for current theories is that on tasks that feature unbalanced experience, rats paradoxically replay the less-experienced trajectory. To understand why, we simulated a feedforward neural network with two regimes: rich learning (structured representations tailored to task demands) and lazy learning (unstructured, task-agnostic representations). Rich, but not lazy, representations degraded following unbalanced experience, an effect that could be reversed with paradoxical replay. To test if this computational principle can account for the experimental data, we examined the relationship between paradoxical replay and learned task representations in the rat hippocampus. Strikingly, we found a strong association between the richness of learned task representations and the paradoxicality of replay. Taken together, these results suggest that paradoxical replay specifically serves to protect rich representations from the destructive effects of unbalanced experience, and more generally demonstrate a novel interaction between the nature of task representations and the function of replay in artificial and biological systems.

Disruption of awake sharp-wave ripples does not affect memorization of locations in repeated-acquisition spatial memory tasks

Storing and accessing memories is required to successfully perform day-to-day tasks, for example for engaging in a meaningful conversation. Previous studies in both rodents and primates have correlated hippocampal cellular activity with behavioral expression of memory. A key role has been attributed to awake hippocampal replay - a sequential reactivation of neurons representing a trajectory through space. However, it is unclear if awake replay impacts immediate future behavior, gradually creates and stabilizes long-term memories over a long period of time (hours and longer), or enables the temporary memorization of relevant events at an intermediate time scale (seconds to minutes). In this study, we aimed to address the uncertainty around the timeframe of impact of awake replay by collecting causal evidence from behaving rats. We detected and disrupted sharp wave ripples (SWRs) - signatures of putative replay events - using electrical stimulation of the ventral hippocampal commissure in rats that were trained on three different spatial memory tasks. In each task, rats were required to memorize a new set of locations in each trial or each daily session. Interestingly, the rats performed equally well with or without SWR disruptions. These data suggest that awake SWRs - and potentially replay - does not affect the immediate behavior nor the temporary memorization of relevant events at a short timescale that are required to successfully perform the spatial tasks. Based on these results, we hypothesize that the impact of awake replay on memory and behavior is long-term and cumulative over time.

Prioritizing replay when future goals are unknown

Although hippocampal place cells replay nonlocal trajectories, the computational function of these events remains controversial. One hypothesis, formalized in a prominent reinforcement learning account, holds that replay plans routes to current goals. However, recent puzzling data appear to contradict this perspective by showing that replayed destinations lag current goals. These results may support an alternative hypothesis that replay updates route information to build a "cognitive map." Yet no similar theory exists to formalize this view, and it is unclear how such a map is represented or what role replay plays in computing it. We address these gaps by introducing a theory of replay that learns a map of routes to candidate goals, before reward is available or when its location may change. Our work extends the planning account to capture a general map-building function for replay, reconciling it with data, and revealing an unexpected relationship between the seemingly distinct hypotheses.

Internal-state-dependent control of feeding behavior via hippocampal ghrelin signaling

Hunger is an internal state that not only invigorates feeding but also acts as a contextual cue for higher-order control of anticipatory feeding-related behavior. The ventral hippocampus is crucial for differentiating optimal behavior across contexts, but how internal contexts such as hunger influence hippocampal circuitry is unknown. In this study, we investigated the role of the ventral hippocampus during feeding behavior across different states of hunger in mice. We found that activity of a unique subpopulation of neurons that project to the nucleus accumbens (vS-NAc neurons) increased when animals investigated food, and this activity inhibited the transition to begin eating. Increases in the level of the peripheral hunger hormone ghrelin reduced vS-NAc activity during this anticipatory phase of feeding via ghrelin-receptor-dependent increases in postsynaptic inhibition and promoted the initiation of eating. Together, these experiments define a ghrelin-sensitive hippocampal circuit that informs the decision to eat based on internal state.

The role of experience in prioritizing hippocampal replay

During sleep, recent memories are replayed by the hippocampus, leading to their consolidation, with a higher priority given to salient experiences. To examine the role of replay in the selective strengthening of memories, we recorded large ensembles of hippocampal place cells while male rats ran repeated spatial trajectories on two linear tracks, differing in either their familiarity or number of laps run. We observed that during sleep, the rate of replay events for a given track increased proportionally with the number of spatial trajectories run by the animal. In contrast, the rate of sleep replay events decreased if the animal was more familiar with the track. Furthermore, we find that the cumulative number of awake replay events occurring during behavior, influenced by both the novelty and duration of an experience, predicts which memories are prioritized for sleep replay, providing a more parsimonious neural correlate for the selective strengthening of memories.

The role of the hippocampus in the consolidation of emotional memories during sleep

Episodic memory relies on the hippocampus, a heterogeneous brain region with distinct functions. Spatial representations in the dorsal hippocampus (dHPC) are crucial for contextual memory, while the ventral hippocampus (vHPC) is more involved in emotional processing. Here, we review the literature in rodents highlighting the anatomical and functional properties of the hippocampus along its dorsoventral axis that underlie its role in contextual and emotional memory encoding, consolidation, and retrieval. We propose that the coordination between the dorsal and vHPC through theta oscillations during rapid eye movement (REM) sleep, and through sharp-wave ripples during non-REM (NREM) sleep, might facilitate the transfer of contextual information for integration with valence-related processing in other structures of the network. Further investigation into the physiology of the vHPC and its connections with other brain areas is needed to deepen the current understanding of emotional memory consolidation during sleep.

Associative and predictive hippocampal codes support memory-guided behaviors

Episodic memory involves learning and recalling associations between items and their spatiotemporal context. Those memories can be further used to generate internal models of the world that enable predictions to be made. The mechanisms that support these associative and predictive aspects of memory are not yet understood. In this study, we used an optogenetic manipulation to perturb the sequential structure, but not global network dynamics, of place cells as rats traversed specific spatial trajectories. This perturbation abolished replay of those trajectories and the development of predictive representations, leading to impaired learning of new optimal trajectories during memory-guided navigation. However, place cell assembly reactivation and reward-context associative learning were unaffected. Our results show a mechanistic dissociation between two complementary hippocampal codes: an associative code (through coactivity) and a predictive code (through sequences).

A map of spatial navigation for neuroscience

Spatial navigation has received much attention from neuroscientists, leading to the identification of key brain areas and the discovery of numerous spatially selective cells. Despite this progress, our understanding of how the pieces fit together to drive behavior is generally lacking. We argue that this is partly caused by insufficient communication between behavioral and neuroscientific researchers. This has led the latter to under-appreciate the relevance and complexity of spatial behavior, and to focus too narrowly on characterizing neural representations of space-disconnected from the computations these representations are meant to enable. We therefore propose a taxonomy of navigation processes in mammals that can serve as a common framework for structuring and facilitating interdisciplinary research in the field. Using the taxonomy as a guide, we review behavioral and neural studies of spatial navigation. In doing so, we validate the taxonomy and showcase its usefulness in identifying potential issues with common experimental approaches, designing experiments that adequately target particular behaviors, correctly interpreting neural activity, and pointing to new avenues of research.

Reap while you sleep: Consolidation of memories differs by how they were sown

Newly formed memories are spontaneously reactivated during sleep, leading to their strengthening. This reactivation process can be manipulated by reinstating learning-related stimuli during sleep, a technique termed targeted memory reactivation. Numerous studies have found that delivering cues during sleep improves memory for simple associations, in which one cue reactivates one tested memory. However, real-life memories often live in rich, complex networks of associations. In this review, we will examine recent forays into investigating how targeted sleep reactivation affects memories within complex paradigms, in which one cue can reactivate multiple tested memories. A common theme across studies is that reactivation consequences do not merely depend on whether memories reside in complex arrangements, but on how memories interact with one another during acquisition. We therefore emphasize how intricate study design details that alter the nature of learning and/or participant intentions impact the outcomes of sleep reactivation. In some cases, complex networks of memories interact harmoniously to bring about mutual memory benefits; in other cases, memories interact antagonistically and produce selective impairments in retrieval. Ultimately, although this burgeoning area of research has yet to be systematically explored, results suggest that the fate of reactivated stimuli within complex arrangements depends on how they were learned.

Spatiotemporal Precision of Neuroimaging in Psychiatry

Aberrant patterns of cognition, perception, and behavior seen in psychiatric disorders are thought to be driven by a complex interplay of neural processes that evolve at a rapid temporal scale. Understanding these dynamic processes in vivo in humans has been hampered by a trade-off between spatial and temporal resolutions inherent to current neuroimaging technology. A recent trend in psychiatric research has been the use of high temporal resolution imaging, particularly magnetoencephalography, often in conjunction with sophisticated machine learning decoding techniques. Developments here promise novel insights into the spatiotemporal dynamics of cognitive phenomena, including domains relevant to psychiatric illnesses such as reward and avoidance learning, memory, and planning. This review considers recent advances afforded by exploiting this increased spatiotemporal precision, with specific reference to applications that seek to drive a mechanistic understanding of psychopathology and the realization of preclinical translation.

Sleep-A brain-state serving systems memory consolidation

Although long-term memory consolidation is supported by sleep, it is unclear how it differs from that during wakefulness. Our review, focusing on recent advances in the field, identifies the repeated replay of neuronal firing patterns as a basic mechanism triggering consolidation during sleep and wakefulness. During sleep, memory replay occurs during slow-wave sleep (SWS) in hippocampal assemblies together with ripples, thalamic spindles, neocortical slow oscillations, and noradrenergic activity. Here, hippocampal replay likely favors the transformation of hippocampus-dependent episodic memory into schema-like neocortical memory. REM sleep following SWS might balance local synaptic rescaling accompanying memory transformation with a sleep-dependent homeostatic process of global synaptic renormalization. Sleep-dependent memory transformation is intensified during early development despite the immaturity of the hippocampus. Overall, beyond its greater efficacy, sleep consolidation differs from wake consolidation mainly in that it is supported, rather than impaired, by spontaneous hippocampal replay activity possibly gating memory formation in neocortex.

Differential replay of reward and punishment paths predicts approach and avoidance

Neural replay is implicated in planning, where states relevant to a task goal are rapidly reactivated in sequence. It remains unclear whether, during planning, replay relates to an actual prospective choice. Here, using magnetoencephalography (MEG), we studied replay in human participants while they planned to either approach or avoid an uncertain environment containing paths leading to reward or punishment. We find evidence for forward sequential replay during planning, with rapid state-to-state transitions from 20 to 90 ms. Replay of rewarding paths was boosted, relative to aversive paths, before a decision to avoid and attenuated before a decision to approach. A trial-by-trial bias toward replaying prospective punishing paths predicted irrational decisions to approach riskier environments, an effect more pronounced in participants with higher trait anxiety. The findings indicate a coupling of replay with planned behavior, where replay prioritizes an online representation of a worst-case scenario for approaching or avoiding.

Enhanced Reactivation of Remapping Place Cells during Aversive Learning

Study of the hippocampal place cell system has greatly enhanced our understanding of memory encoding for distinct places, but how episodic memories for distinct experiences occurring within familiar environments are encoded is less clear. We developed a spatial decision-making task in which male rats learned to navigate a multiarm maze to a goal location for food reward while avoiding maze arms in which aversive stimuli were delivered. Task learning induced partial remapping in CA1 place cells, allowing us to identify both remapping and stable cell populations. Remapping cells were recruited into sharp-wave ripples and associated replay events to a greater extent than stable cells, despite having similar firing rates during navigation of the maze. Our results suggest that recruitment into replay events may be a mechanism to incorporate new contextual information into a previously formed and stabilized spatial representation.SIGNIFICANCE STATEMENT Hippocampal place cells provide a map of space that animals use to navigate. This map can change to reflect changes in the physical properties of the environment in which the animal finds itself, and also in response to nonphysical contextual changes, such as changes in the valence of specific locations within that environment. We show here that cells which change their spatial tuning after a change in context are preferentially recruited into sharp-wave ripple-associated replay events compared with stable nonremapping cells. Thus, our data lend strong support to the hypothesis that replay is a mechanism for the storage of new spatial maps.

Heterogeneity of network and coding states in mouse CA1 place cells

Theta sequences and phase precession shape hippocampal activity and are considered key underpinnings of memory formation. Theta sequences are sweeps of spikes from multiple cells, tracing trajectories from past to future. Phase precession is the correlation between theta firing phase and animal position. Here, we reconsider these temporal processes in CA1 and the computational principles that they are thought to obey. We find stronger heterogeneity than previously described: we identify cells that do not phase precess but reliably express theta sequences. Other cells phase precess only when medium gamma (linked to entorhinal inputs) is strongest. The same cells express more sequences, but not precession, when slow gamma (linked to CA3 inputs) dominates. Moreover, sequences occur independently in distinct cell groups. Our results challenge the view that phase precession is the mechanism underlying the emergence of theta sequences, suggesting a role for CA1 cells in multiplexing diverse computational processes.

Imagination as a fundamental function of the hippocampus

Imagination is a biological function that is vital to human experience and advanced cognition. Despite this importance, it remains unknown how imagination is realized in the brain. Substantial research focusing on the hippocampus, a brain structure traditionally linked to memory, indicates that firing patterns in spatially tuned neurons can represent previous and upcoming paths in space. This work has generally been interpreted under standard views that the hippocampus implements cognitive abilities primarily related to actual experience, whether in the past (e.g. recollection, consolidation), present (e.g. spatial mapping) or future (e.g. planning). However, relatively recent findings in rodents identify robust patterns of hippocampal firing corresponding to a variety of alternatives to actual experience, in many cases without overt reference to the past, present or future. Given these findings, and others on hippocampal contributions to human imagination, we suggest that a fundamental function of the hippocampus is to generate a wealth of hypothetical experiences and thoughts. Under this view, traditional accounts of hippocampal function in episodic memory and spatial navigation can be understood as particular applications of a more general system for imagination. This view also suggests that the hippocampus contributes to a wider range of cognitive abilities than previously thought. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

CA2 physiology underlying social memory

In recent years, convergent evidence has emerged in support of the idea of social brain networks, specific brain regions that are interconnected and support social behaviors. One of these regions is the CA2 area of the hippocampus, a small region strongly connected with cortical and subcortical areas implicated in social behaviors. Furthermore, CA2 area is enriched in receptors for several neuromodulators that are related to various aspects of social behaviors, suggesting that this area could be a key component of social information processing in the brain. In this review, recent findings related to the physiological mechanisms underlying the role of CA2 in social memory are discussed.

Experience-driven rate modulation is reinstated during hippocampal replay

Replay, the sequential reactivation within a neuronal ensemble, is a central hippocampal mechanism postulated to drive memory processing. While both rate and place representations are used by hippocampal place cells to encode behavioral episodes, replay has been largely defined by only the latter – based on the fidelity of sequential activity across neighboring place fields. Here, we show that dorsal CA1 place cells in rats can modulate their firing rate between replay events of two different contexts. This experience-dependent phenomenon mirrors the same pattern of rate modulation observed during behavior and can be used independently from place information within replay sequences to discriminate between contexts. Our results reveal the existence of two complementary neural representations available for memory processes.

How do our brains store memories? We now know that this is a complex and dynamic process, involving multiple regions of the brain. A brain region, called the hippocampus, plays an important role in memory formation. While we sleep, the hippocampus works to consolidate information, and eventually creates stable, long-term memories that are then stored in other parts of the brain.

But how does the hippocampus do this? Neuroscientists believe that it can replay the patterns of brain activity that represent particular memories. By repeatedly doing this while we sleep, the hippocampus can then direct the transfer of this information to the rest of the brain for storage.

The behaviour of nerve cells in the brain underpins these patterns of brain activity. When a nerve cell is active, it fires tiny electrical impulses that can be detected experimentally. The brain thus represents information in two ways: which nerve cells are active and when (sequential patterns); and how active the nerve cells are (how fast they fire electrical impulses or firing rate). For example, when an animal moves from one location to another, special place cells in the hippocampus become active in a distinct sequence. Depending on the context, they will also fire faster or slower.

We know that the hippocampus can replay sequential patterns of nerve cell activity during memory consolidation, but whether it can also replay the firing rates associated with a particular experience is still unknown. Tirole, Huelin Gorriz et al. set out to determine if the hippocampus could also preserve the information encoded by firing rate during replay.

In the experiments, rats explored two different environments that they had not seen before. The activity of the rats’ place cells was recorded before and after they explored, and also later while they were sleeping. Analysis of the recordings revealed that during replay, the rats’ hippocampi could indeed reproduce both the sequential patterns of activity and the firing rate of the place cells. It also confirmed that each environment was associated with unique firing rates – in other words, the firing rates were memory-specific.

These results contribute to our understanding of how the hippocampus represents and processes information about our experiences. More broadly, they also shed new light on how the brain lays down memories, by revealing a key part of the mechanism that it uses to consolidate that information.

Multiple-Timescale Representations of Space: Linking Memory to Navigation

When navigating through space, we must maintain a representation of our position in real time; when recalling a past episode, a memory can come back in a flash. Interestingly, the brain's spatial representation system, including the hippocampus, supports these two distinct timescale functions. How are neural representations of space used in the service of both real-world navigation and internal mnemonic processes? Recent progress has identified sequences of hippocampal place cells, evolving at multiple timescales in accordance with either navigational behaviors or internal oscillations, that underlie these functions. We review experimental findings on experience-dependent modulation of these sequential representations and consider how they link real-world navigation to time-compressed memories. We further discuss recent work suggesting the prevalence of these sequences beyond hippocampus and propose that these multiple-timescale mechanisms may represent a general algorithm for organizing cell assemblies, potentially unifying the dual roles of the spatial representation system in memory and navigation.

Observational learning promotes hippocampal remote awake replay toward future reward locations

The neural circuit mechanisms underlying observational learning, learning through observing the behavior of others, are poorly understood. Hippocampal place cells are important for spatial learning, and awake replay of place cell patterns is involved in spatial decisions. Here we show that, in observer rats learning to run a maze by watching a demonstrator's spatial trajectories from a separate nearby observation box, place cell patterns during self-running in the maze are replayed remotely in the box. The contents of the remote awake replay preferentially target the maze's reward sites from both forward and reverse replay directions and reflect the observer's future correct trajectories in the maze. In contrast, under control conditions without a demonstrator, the remote replay is significantly reduced, and the preferences for reward sites and future trajectories disappear. Our results suggest that social observation directs the contents of remote awake replay to guide spatial decisions in observational learning.

Planning in the brain

Recent breakthroughs in artificial intelligence (AI) have enabled machines to plan in tasks previously thought to be uniquely human. Meanwhile, the planning algorithms implemented by the brain itself remain largely unknown. Here, we review neural and behavioral data in sequential decision-making tasks that elucidate the ways in which the brain does-and does not-plan. To systematically review available biological data, we create a taxonomy of planning algorithms by summarizing the relevant design choices for such algorithms in AI. Across species, recording techniques, and task paradigms, we find converging evidence that the brain represents future states consistent with a class of planning algorithms within our taxonomy-focused, depth-limited, and serial. However, we argue that current data are insufficient for addressing more detailed algorithmic questions. We propose a new approach leveraging AI advances to drive experiments that can adjudicate between competing candidate algorithms.

Spatial goal coding in the hippocampal formation

The mammalian hippocampal formation contains several distinct populations of neurons involved in representing self-position and orientation. These neurons, which include place, grid, head direction, and boundary-vector cells, are thought to collectively instantiate cognitive maps supporting flexible navigation. However, to flexibly navigate, it is necessary to also maintain internal representations of goal locations, such that goal-directed routes can be planned and executed. Although it has remained unclear how the mammalian brain represents goal locations, multiple neural candidates have recently been uncovered during different phases of navigation. For example, during planning, sequential activation of spatial cells may enable simulation of future routes toward the goal. During travel, modulation of spatial cells by the prospective route, or by distance and direction to the goal, may allow maintenance of route and goal-location information, supporting navigation on an ongoing basis. As the goal is approached, an increased activation of spatial cells may enable the goal location to become distinctly represented within cognitive maps, aiding goal localization. Lastly, after arrival at the goal, sequential activation of spatial cells may represent the just-taken route, enabling route learning and evaluation. Here, we review and synthesize these and other evidence for goal coding in mammalian brains, relate the experimental findings to predictions from computational models, and discuss outstanding questions and future challenges.

Reactivation predicts the consolidation of unbiased long-term cognitive maps

Spatial memories that can last a lifetime are thought to be encoded during 'online' periods of exploration and subsequently consolidated into stable cognitive maps through their 'offline' reactivation. However, the mechanisms and computational principles by which offline reactivation stabilize long-lasting spatial representations remain poorly understood. Here, we employed simultaneous fast calcium imaging and electrophysiology to track hippocampal place cells over 2 weeks of online spatial reward learning behavior and offline resting. We describe that recruitment to persistent network-level offline reactivation of spatial experiences in mice predicts the future representational stability of place cells days in advance of their online reinstatement. Moreover, while representations of reward-adjacent locations are generally more stable across days, offline-reactivation-related stability is, conversely, most prominent for reward-distal locations. Thus, while occurring on the tens of milliseconds timescale, offline reactivation is uniquely associated with the stability of multiday representations that counterbalance the overall reward-adjacency bias, thereby predicting the stabilization of cognitive maps that comprehensively reflect entire underlying spatial contexts. These findings suggest that post-learning offline-related memory consolidation plays a complimentary and computationally distinct role in learning compared to online encoding.

The orbitofrontal cortex maps future navigational goals

Accurate navigation to a desired goal requires consecutive estimates of spatial relationships between the current position and future destination throughout the journey. Although neurons in the hippocampal formation can represent the position of an animal as well as its nearby trajectories^1-7, their role in determining the destination of the animal has been questioned^8,9. It is, thus, unclear whether the brain can possess a precise estimate of target location during active environmental exploration. Here we describe neurons in the rat orbitofrontal cortex (OFC) that form spatial representations persistently pointing to the subsequent goal destination of an animal throughout navigation. This destination coding emerges before the onset of navigation, without direct sensory access to a distal goal, and even predicts the incorrect destination of an animal at the beginning of an error trial. Goal representations in the OFC are maintained by destination-specific neural ensemble dynamics, and their brief perturbation at the onset of a journey led to a navigational error. These findings suggest that the OFC is part of the internal goal map of the brain, enabling animals to navigate precisely to a chosen destination that is beyond the range of sensory perception.

Between-subject prediction reveals a shared representational geometry in the rodent hippocampus

The rodent hippocampus constructs statistically independent representations across environments ("global remapping") and assigns individual neuron firing fields to locations within an environment in an apparently random fashion, processes thought to contribute to the role of the hippocampus in episodic memory. This random mapping implies that it should be challenging to predict hippocampal encoding of a given experience in one subject based on the encoding of that same experience in another subject. Contrary to this prediction, we find that by constructing a common representational space across rats in which neural activity is aligned using geometric operations (rotation, reflection, and translation; "hyperalignment"), we can predict data of "right" trials (R) on a T-maze in a target rat based on (1) the "left" trials (L) of the target rat and (2) the relationship between L and R trials from a different source rat. These cross-subject predictions relied on ensemble activity patterns, including both firing rate and field location, and outperformed a number of control mappings, such as those based on permuted data that broke the relationship between L and R activity for individual neurons and those based solely on within-subject prediction. This work constitutes proof of principle for successful cross-subject prediction of ensemble activity patterns in the hippocampus and provides new insights in understanding how different experiences are structured, enabling further work identifying what aspects of experience encoding are shared versus unique to an individual.

Hippocampal replay reflects specific past experiences rather than a plan for subsequent choice

Executing memory-guided behavior requires storage of information about experience and later recall of that information to inform choices. Awake hippocampal replay, when hippocampal neural ensembles briefly reactivate a representation related to prior experience, has been proposed to critically contribute to these memory-related processes. However, it remains unclear whether awake replay contributes to memory function by promoting the storage of past experiences, facilitating planning based on evaluation of those experiences, or both. We designed a dynamic spatial task that promotes replay before a memory-based choice and assessed how the content of replay related to past and future behavior. We found that replay content was decoupled from subsequent choice and instead was enriched for representations of previously rewarded locations and places that had not been visited recently, indicating a role in memory storage rather than in directly guiding subsequent behavior.

Hippocampal sharp-wave ripples in cognitive map maintenance versus episodic simulation

Hippocampal sharp-wave ripples (SWRs) have been proposed to support memory-based decision-making. A new study by Gillespie et al. (in this issue of Neuron) provides important new insights on how past experiences and future choices are reflected in neuronal activity during SWRs.

Hippocampal replay of experience at real-world speeds

Representations related to past experiences play a critical role in memory and decision-making processes. The rat hippocampus expresses these types of representations during sharp-wave ripple (SWR) events, and previous work identified a minority of SWRs that contain ‘replay’ of spatial trajectories at ∼20x the movement speed of the animal. Efforts to understand replay typically make multiple assumptions about which events to examine and what sorts of representations constitute replay. We therefore lack a clear understanding of both the prevalence and the range of representational dynamics associated with replay. Here, we develop a state space model that uses a combination of movement dynamics of different speeds to capture the spatial content and time evolution of replay during SWRs. Using this model, we find that the large majority of replay events contain spatially coherent, interpretable content. Furthermore, many events progress at real-world, rather than accelerated, movement speeds, consistent with actual experiences.

Learning offline: memory replay in biological and artificial reinforcement learning

Learning to act in an environment to maximise rewards is among the brain's key functions. This process has often been conceptualised within the framework of reinforcement learning, which has also gained prominence in machine learning and artificial intelligence (AI) as a way to optimise decision making. A common aspect of both biological and machine reinforcement learning is the reactivation of previously experienced episodes, referred to as replay. Replay is important for memory consolidation in biological neural networks and is key to stabilising learning in deep neural networks. Here, we review recent developments concerning the functional roles of replay in the fields of neuroscience and AI. Complementary progress suggests how replay might support learning processes, including generalisation and continual learning, affording opportunities to transfer knowledge across the two fields to advance the understanding of biological and artificial learning and memory.

Replay in minds and machines

Experience-related brain activity patterns reactivate during sleep, wakeful rest, and brief pauses from active behavior. In parallel, machine learning research has found that experience replay can lead to substantial performance improvements in artificial agents. Together, these lines of research suggest replay has a variety of computational benefits for decision-making and learning. Here, we provide an overview of putative computational functions of replay as suggested by machine learning and neuroscientific research. We show that replay can lead to faster learning, less forgetting, reorganization or augmentation of experiences, and support planning and generalization. In addition, we highlight the benefits of reactivating abstracted internal representations rather than veridical memories, and discuss how replay could provide a mechanism to build internal representations that improve learning and decision-making.

Navigating for reward

An organism's survival can depend on its ability to recall and navigate to spatial locations associated with rewards, such as food or a home. Accumulating research has revealed that computations of reward and its prediction occur on multiple levels across a complex set of interacting brain regions, including those that support memory and navigation. However, how the brain coordinates the encoding, recall and use of reward information to guide navigation remains incompletely understood. In this Review, we propose that the brain's classical navigation centres - the hippocampus and the entorhinal cortex - are ideally suited to coordinate this larger network by representing both physical and mental space as a series of states. These states may be linked to reward via neuromodulatory inputs to the hippocampus-entorhinal cortex system. Hippocampal outputs can then broadcast sequences of states to the rest of the brain to store reward associations or to facilitate decision-making, potentially engaging additional value signals downstream. This proposal is supported by recent advances in both experimental and theoretical neuroscience. By discussing the neural systems traditionally tied to navigation and reward at their intersection, we aim to offer an integrated framework for understanding navigation to reward as a fundamental feature of many cognitive processes.

Disrupting the medial prefrontal cortex with designer receptors exclusively activated by designer drug alters hippocampal sharp-wave ripples and their associated cognitive processes

The hippocampus and medial prefrontal cortex (mPFC) interact during a myriad of cognitive processes including decision-making and long-term memory consolidation. Exactly how the mPFC and hippocampus interact during goal-directed decision-making remains to be fully elucidated. During periods of rest, bursts of high-frequency oscillations, termed sharp-wave ripple (SWR), appear in the local field potential. Impairing SWRs on the maze or during post-learning rest can interfere with memory-guided decision-making and memory consolidation. We hypothesize that the hippocampus and mPFC bidirectionally interact during SWRs to support memory consolidation and decision-making. Rats were trained on the neuroeconomic spatial decision-making task, Restaurant Row, to make serial stay-skip decisions where the amount of effort (delay to reward) varied upon entry to each restaurant. Hippocampal cells and SWRs were recorded in rats with the mPFC transduced with inhibitory DREADDs. We found that disrupting the mPFC impaired consolidating SWRs in the hippocampus. Hippocampal SWR rates depended on the internalized value of the reward (derived from individual flavor preferences), a parameter important in decision-making, and disrupting the mPFC changed this relationship. Additionally, we found a dissociation between SWRs that occurred while rats were on the maze dependent upon whether those SWRs occurred while the rat was anticipating food reward or during post-reward consumption.

The evolving view of replay and its functions in wake and sleep

The term hippocampal replay originally referred to the temporally compressed reinstantiation, during rest, of sequential neural activity observed during prior active wake. Since its description in the 1990s, hippocampal replay has often been viewed as the key mechanism by which a memory trace is repeatedly rehearsed at high speeds during sleep and gradually transferred to neocortical circuits. However, the methods used to measure the occurrence of replay remain debated, and it is now clear that the underlying neural events are considerably more complicated than the traditional narratives had suggested. “Replay-like” activity happens during wake, can play out in reverse order, may represent trajectories never taken by the animal, and may have additional functions beyond memory consolidation, from learning values and solving the problem of credit assignment to decision-making and planning. Still, we know little about the role of replay in cognition, and to what extent it differs between wake and sleep. This may soon change, however, because decades-long efforts to explain replay in terms of reinforcement learning (RL) have started to yield testable predictions and possible explanations for a diverse set of observations. Here, we (1) survey the diverse features of replay, focusing especially on the latest findings; (2) discuss recent attempts at unifying disparate experimental results and putatively different cognitive functions under the banner of RL; (3) discuss methodological issues and theoretical biases that impede progress or may warrant a partial revaluation of the current literature, and finally; (4) highlight areas of considerable uncertainty and promising avenues of inquiry.

Prioritized experience replays on a hippocampal predictive map for learning

The hippocampus is crucial for spatial learning but their neuronal mechanisms remain unknown. Here, we found that hippocampal cell ensembles preferentially replayed salient reward-related locations as rats learned a new behavioral trajectory for reward. The contents of these hippocampal replays progressively varied with learning. Especially during learning, hippocampal neurons even replayed an optimized path that had never been exploited by the animals. The learning-related hippocampal activity was necessary for the stabilization of spatial behavior. These results suggest prioritized replays of significant experiences on a predictive map by hippocampal neurons.

Hippocampal cells are central to spatial and predictive representations, and experience replays by place cells are crucial for learning and memory. Nonetheless, how hippocampal replay patterns dynamically change during the learning process remains to be elucidated. Here, we designed a spatial task in which rats learned a new behavioral trajectory for reward. We found that as rats updated their behavioral strategies for a novel salient location, hippocampal cell ensembles increased theta-sequences and sharp wave ripple-associated synchronous spikes that preferentially replayed salient locations and reward-related contexts in reverse order. The directionality and contents of the replays progressively varied with learning, including an optimized path that had never been exploited by the animals, suggesting prioritized replays of significant experiences on a predictive map. Online feedback blockade of sharp wave ripples during a learning process inhibited stabilizing optimized behavior. These results implicate learning-associated experience replays that act to learn and reinforce specific behavioral strategies.

Reward-predictive representations generalize across tasks in reinforcement learning

In computer science, reinforcement learning is a powerful framework with which artificial agents can learn to maximize their performance for any given Markov decision process (MDP). Advances over the last decade, in combination with deep neural networks, have enjoyed performance advantages over humans in many difficult task settings. However, such frameworks perform far less favorably when evaluated in their ability to generalize or transfer representations across different tasks. Existing algorithms that facilitate transfer typically are limited to cases in which the transition function or the optimal policy is portable to new contexts, but achieving "deep transfer" characteristic of human behavior has been elusive. Such transfer typically requires discovery of abstractions that permit analogical reuse of previously learned representations to superficially distinct tasks. Here, we demonstrate that abstractions that minimize error in predictions of reward outcomes generalize across tasks with different transition and reward functions. Such reward-predictive representations compress the state space of a task into a lower dimensional representation by combining states that are equivalent in terms of both the transition and reward functions. Because only state equivalences are considered, the resulting state representation is not tied to the transition and reward functions themselves and thus generalizes across tasks with different reward and transition functions. These results contrast with those using abstractions that myopically maximize reward in any given MDP and motivate further experiments in humans and animals to investigate if neural and cognitive systems involved in state representation perform abstractions that facilitate such equivalence relations.

The roles of online and offline replay in planning

Animals and humans replay neural patterns encoding trajectories through their environment, both whilst they solve decision-making tasks and during rest. Both on-task and off-task replay are believed to contribute to flexible decision making, though how their relative contributions differ remains unclear. We investigated this question by using magnetoencephalography (MEG) to study human subjects while they performed a decision-making task that was designed to reveal the decision algorithms employed. We characterised subjects in terms of how flexibly each adjusted their choices to changes in temporal, spatial and reward structure. The more flexible a subject, the more they replayed trajectories during task performance, and this replay was coupled with re-planning of the encoded trajectories. The less flexible a subject, the more they replayed previously preferred trajectories during rest periods between task epochs. The data suggest that online and offline replay both participate in planning but support distinct decision strategies.

Potential factors influencing replay across CA1 during sharp-wave ripples

Sharp-wave ripples are complex neurophysiological events recorded along the trisynaptic hippocampal circuit (i.e. from CA3 to CA1 and the subiculum) during slow-wave sleep and awake states. They arise locally but scale brain-wide to the hippocampal target regions at cortical and subcortical structures. During these events, neuronal firing sequences are replayed retrospectively or prospectively and in the forward or reverse order as defined by experience. They could reflect either pre-configured firing sequences, learned sequences or an option space to inform subsequent decisions. How can different sequences arise during sharp-wave ripples? Emerging data suggest the hippocampal circuit is organized in different loops across the proximal (close to dentate gyrus) and distal (close to entorhinal cortex) axis. These data also disclose a so-far neglected laminar organization of the hippocampal output during sharp-wave events. Here, I discuss whether by incorporating cell-type-specific mechanisms converging on deep and superficial CA1 sublayers along the proximodistal axis, some novel factors influencing the organization of hippocampal sequences could be unveiled. This article is part of the Theo Murphy meeting issue 'Memory reactivation: replaying events past, present and future'.

Progress and issues in second-order analysis of hippocampal replay

Patterns of neural activity that occur spontaneously during sharp-wave ripple (SWR) events in the hippocampus are thought to play an important role in memory formation, consolidation and retrieval. Typical studies examining the content of SWRs seek to determine whether the identity and/or temporal order of cell firing is different from chance. Such 'first-order' analyses are focused on a single time point and template (map), and have been used to show, for instance, the existence of preplay. The major methodological challenge in first-order analyses is the construction and interpretation of different chance distributions. By contrast, 'second-order' analyses involve a comparison of SWR content between different time points, and/or between different templates. Typical second-order questions include tests of experience-dependence (replay) that compare SWR content before and after experience, and comparisons or replay between different arms of a maze. Such questions entail additional methodological challenges that can lead to biases in results and associated interpretations. We provide an inventory of analysis challenges for second-order questions about SWR content, and suggest ways of preventing, identifying and addressing possible analysis biases. Given evolving interest in understanding SWR content in more complex experimental scenarios and across different time scales, we expect these issues to become increasingly pervasive. This article is part of the Theo Murphy meeting issue 'Memory reactivation: replaying events past, present and future'.