Distinct effects of reward and navigation history on hippocampal forward and reverse replays

Hippocampal representations of alternative possibilities are flexibly generated to meet cognitive demands

The cognitive ability to go beyond the present to consider alternative possibilities, including potential futures and counterfactual pasts, can support adaptive decision making. Complex and changing real-world environments, however, have many possible alternatives. Whether and how the brain can select among them to represent alternatives that meet current cognitive needs remains unknown. We therefore examined neural representations of alternative spatial locations in the rat hippocampus during navigation in a complex patch foraging environment with changing reward probabilities. We found representations of multiple alternatives along paths ahead and behind the animal, including in distant alternative patches. Critically, these representations were modulated in distinct patterns across successive trials: alternative paths were represented proportionate to their evolving relative value and predicted subsequent decisions, whereas distant alternatives were prevalent during value updating. These results demonstrate that the brain modulates the generation of alternative possibilities in patterns that meet changing cognitive needs for adaptive behavior.

Selective reactivation of value- and place-dependent information during sharp-wave ripples in the intermediate and dorsal hippocampus

Reactivating place cells during sharp-wave ripples in the hippocampus is important for memory consolidation. However, whether hippocampal reactivation is affected by the values of events experienced by the animal is largely unknown. Here, we investigated whether place cells in the dorsal (dHP) and intermediate hippocampus (iHP) of rats are differentially reactivated depending on the value associated with a place during the learning of places associated with higher-value rewards in a T-maze. Place cells in the iHP representing the high-value location were reactivated significantly more frequently than those representing the low-value location, characteristics not observed in the dHP. In contrast, the activities of place cells in the dHP coding the routes leading to high-value locations were replayed more than those in the iHP. Our findings suggest that value-based differential reactivation patterns along the septotemporal axis of the hippocampus may play essential roles in optimizing goal-directed spatial learning for maximal reward.

Efficient, continual, and generalized learning in the brain - neural mechanism of Mental Schema 2.0

There has been tremendous progress in artificial neural networks (ANNs) over the past decade; however, the gap between ANNs and the biological brain as a learning device remains large. With the goal of closing this gap, this paper reviews learning mechanisms in the brain by focusing on three important issues in ANN research: efficiency, continuity, and generalization. We first discuss the method by which the brain utilizes a variety of self-organizing mechanisms to maximize learning efficiency, with a focus on the role of spontaneous activity of the brain in shaping synaptic connections to facilitate spatiotemporal learning and numerical processing. Then, we examined the neuronal mechanisms that enable lifelong continual learning, with a focus on memory replay during sleep and its implementation in brain-inspired ANNs. Finally, we explored the method by which the brain generalizes learned knowledge in new situations, particularly from the mathematical generalization perspective of topology. Besides a systematic comparison in learning mechanisms between the brain and ANNs, we propose "Mental Schema 2.0," a new computational property underlying the brain's unique learning ability that can be implemented in ANNs.

Sensory cortical ensembles exhibit differential coupling to ripples in distinct hippocampal subregions

Cortical neurons activated during recent experiences often reactivate with dorsal hippocampal CA1 ripples during subsequent rest. Less is known about cortical interactions with intermediate hippocampal CA1, whose connectivity, functions, and ripple events differ from dorsal CA1. We identified three clusters of putative excitatory neurons in mouse visual cortex that are preferentially excited together with either dorsal or intermediate CA1 ripples or suppressed before both ripples. Neurons in each cluster were evenly distributed across primary and higher visual cortices and co-active even in the absence of ripples. These ensembles exhibited similar visual responses but different coupling to thalamus and pupil-indexed arousal. We observed a consistent activity sequence preceding and predicting ripples: (1) suppression of ripple-suppressed cortical neurons, (2) thalamic silence, and (3) activation of intermediate CA1-ripple-activated cortical neurons. We propose that coordinated dynamics of these ensembles relay visual experiences to distinct hippocampal subregions for incorporation into different cognitive maps.

Hippocampal place code plasticity in CA1 requires postsynaptic membrane fusion

Rapid delivery of glutamate receptors to the postsynaptic membrane via vesicle fusion is a central component of synaptic plasticity. However, it is unknown how this process supports specific neural computations during behavior. To bridge this gap, we combined conditional genetic deletion of a component of the postsynaptic membrane fusion machinery, Syntaxin3 (Stx3), in hippocampal CA1 neurons of mice with population in vivo calcium imaging. This approach revealed that Stx3 is necessary for forming the neural dynamics that support novelty processing, spatial reward memory and offline memory consolidation. In contrast, CA1 Stx3 was dispensable for maintaining aspects of the neural code that exist presynaptic to CA1 such as representations of context and space. Thus, manipulating postsynaptic membrane fusion identified computations that specifically require synaptic restructuring via membrane trafficking in CA1 and distinguished them from neural representation that could be inherited from upstream brain regions or learned through other mechanisms.

Dual credit assignment processes underlie dopamine signals in a complex spatial environment

Animals frequently make decisions based on expectations of future reward ("values"). Values are updated by ongoing experience: places and choices that result in reward are assigned greater value. Yet, the specific algorithms used by the brain for such credit assignment remain unclear. We monitored accumbens dopamine as rats foraged for rewards in a complex, changing environment. We observed brief dopamine pulses both at reward receipt (scaling with prediction error) and at novel path opportunities. Dopamine also ramped up as rats ran toward reward ports, in proportion to the value at each location. By examining the evolution of these dopamine place-value signals, we found evidence for two distinct update processes: progressive propagation of value along taken paths, as in temporal difference learning, and inference of value throughout the maze, using internal models. Our results demonstrate that within rich, naturalistic environments dopamine conveys place values that are updated via multiple, complementary learning algorithms.

Dual credit assignment processes underlie dopamine signals in a complex spatial environment

Dopamine in the nucleus accumbens helps motivate behavior based on expectations of future reward ("values"). These values need to be updated by experience: after receiving reward, the choices that led to reward should be assigned greater value. There are multiple theoretical proposals for how this credit assignment could be achieved, but the specific algorithms that generate updated dopamine signals remain uncertain. We monitored accumbens dopamine as freely behaving rats foraged for rewards in a complex, changing environment. We observed brief pulses of dopamine both when rats received reward (scaling with prediction error), and when they encountered novel path opportunities. Furthermore, dopamine ramped up as rats ran towards reward ports, in proportion to the value at each location. By examining the evolution of these dopamine place-value signals, we found evidence for two distinct update processes: progressive propagation along taken paths, as in temporal-difference learning, and inference of value throughout the maze, using internal models. Our results demonstrate that within rich, naturalistic environments dopamine conveys place values that are updated via multiple, complementary learning algorithms.

Sensory cortical ensembles exhibit differential coupling to ripples in distinct hippocampal subregions

Cortical neurons activated during recent experiences often reactivate with dorsal hippocampal CA1 sharp-wave ripples (SWRs) during subsequent rest. Less is known about cortical interactions with intermediate hippocampal CA1, whose connectivity, functions, and SWRs differ from those of dorsal CA1. We identified three clusters of visual cortical excitatory neurons that are excited together with either dorsal or intermediate CA1 SWRs, or suppressed before both SWRs. Neurons in each cluster were distributed across primary and higher visual cortices and co-active even in the absence of SWRs. These ensembles exhibited similar visual responses but different coupling to thalamus and pupil-indexed arousal. We observed a consistent activity sequence: (i) suppression of SWR-suppressed cortical neurons, (ii) thalamic silence, and (iii) activation of the cortical ensemble preceding and predicting intermediate CA1 SWRs. We propose that the coordinated dynamics of these ensembles relay visual experiences to distinct hippocampal subregions for incorporation into different cognitive maps.

How our understanding of memory replay evolves

Memory reactivations and replay, widely reported in the hippocampus and cortex across species, have been implicated in memory consolidation, planning, and spatial and skill learning. Technological advances in electrophysiology, calcium imaging, and human neuroimaging techniques have enabled neuroscientists to measure large-scale neural activity with increasing spatiotemporal resolution and have provided opportunities for developing robust analytic methods to identify memory replay. In this article, we first review a large body of historically important and representative memory replay studies from the animal and human literature. We then discuss our current understanding of memory replay functions in learning, planning, and memory consolidation and further discuss the progress in computational modeling that has contributed to these improvements. Next, we review past and present analytic methods for replay analyses and discuss their limitations and challenges. Finally, looking ahead, we discuss some promising analytic methods for detecting nonstereotypical, behaviorally nondecodable structures from large-scale neural recordings. We argue that seamless integration of multisite recordings, real-time replay decoding, and closed-loop manipulation experiments will be essential for delineating the role of memory replay in a wide range of cognitive and motor functions.

Synthetic Data Resource and Benchmarks for Time Cell Analysis and Detection Algorithms

Hippocampal CA1 cells take part in reliable, time-locked activity sequences in tasks that involve an association between temporally separated stimuli, in a manner that tiles the interval between the stimuli. Such cells have been termed time cells. Here, we adopt a first-principles approach to comparing diverse analysis and detection algorithms for identifying time cells. We generated synthetic activity datasets using calcium signals recorded in vivo from the mouse hippocampus using two-photon (2-P) imaging, as template response waveforms. We assigned known, ground truth values to perturbations applied to perfect activity signals, including noise, calcium event width, timing imprecision, hit trial ratio and background (untuned) activity. We tested a range of published and new algorithms and their variants on this dataset. We find that most algorithms correctly classify over 80% of cells, but have different balances between true and false positives, and different sensitivity to the five categories of perturbation. Reassuringly, most methods are reasonably robust to perturbations, including background activity, and show good concordance in classification of time cells. The same algorithms were also used to analyze and identify time cells in experimental physiology datasets recorded in vivo and most show good concordance.

Septotemporal variations in hippocampal value and outcome processing

A large body of evidence indicates functional variations along the hippocampal longitudinal axis. To investigate whether and how value and outcome processing vary between the dorsal (DH) and the ventral hippocampus (VH), we examined neuronal activity and inactivation effects of the DH and VH in mice performing probabilistic classical conditioning tasks. Inactivation of either structure disrupts value-dependent anticipatory licking, and value-coding neurons are found in both structures, indicating their involvement in value processing. However, the DH neuronal population increases activity as a function of value, while the VH neuronal population is preferentially responsive to the highest-value sensory cue. Also, signals related to outcome-dependent value learning are stronger in the DH. VH neurons instead show rapid responses to punishment and strongly biased responses to negative prediction error. These findings suggest that the DH faithfully represents the external value landscape, whereas the VH preferentially represents behaviorally relevant, salient features of experienced events.

Place cells are nonrandomly clustered by field location in CA1 hippocampus

A challenge in both modern and historic neuroscience has been achieving an understanding of neuron circuits, and determining the computational and organizational principles that underlie these circuits. Deeper understanding of the organization of brain circuits and cell types, including in the hippocampus, is required for advances in behavioral and cognitive neuroscience, as well as for understanding principles governing brain development and evolution. In this manuscript, we pioneer a new method to analyze the spatial clustering of active neurons in the hippocampus. We use calcium imaging and a rewarded navigation task to record from 100 s of place cells in the CA1 of freely moving rats. We then use statistical techniques developed for and in widespread use in geographic mapping studies, global Moran's I, and local Moran's I to demonstrate that cells that code for similar spatial locations tend to form small spatial clusters. We present evidence that this clustering is not the result of artifacts from calcium imaging, and show that these clusters are primarily formed by cells that have place fields around previously rewarded locations. We go on to show that, although cells with similar place fields tend to form clusters, there is no obvious topographic mapping of environmental location onto the hippocampus, such as seen in the visual cortex. Insights into hippocampal organization, as in this study, can elucidate mechanisms underlying motivational behaviors, spatial navigation, and memory formation.

Reward expectation extinction restructures and degrades CA1 spatial maps through loss of a dopaminergic reward proximity signal

Hippocampal place cells support reward-related spatial memories by forming a cognitive map that over-represents reward locations. The strength of these memories is modulated by the extent of reward expectation during encoding. However, the circuit mechanisms underlying this modulation are unclear. Here we find that when reward expectation is extinguished in mice, they remain engaged with their environment, yet place cell over-representation of rewards vanishes, place field remapping throughout the environment increases, and place field trial-to-trial reliability decreases. Interestingly, Ventral Tegmental Area (VTA) dopaminergic axons in CA1 exhibit a ramping reward-proximity signal that depends on reward expectation and inhibiting VTA dopaminergic neurons largely replicates the effects of extinguishing reward expectation. We conclude that changing reward expectation restructures CA1 cognitive maps and determines map reliability by modulating the dopaminergic VTA-CA1 reward-proximity signal. Thus, internal states of high reward expectation enhance encoding of spatial memories by reinforcing hippocampal cognitive maps associated with reward.

Internally Triggered Experiences of Hedonic Valence in Nonhuman Animals: Cognitive and Welfare Considerations

Do any nonhuman animals have hedonically valenced experiences not directly caused by stimuli in their current environment? Do they, like us humans, experience anticipated or previously experienced pains and pleasures as respectively painful and pleasurable? We review evidence from comparative neuroscience about hippocampus-dependent simulation in relation to this question. Hippocampal sharp-wave ripples and theta oscillations have been found to instantiate previous and anticipated experiences. These hippocampal activations coordinate with neural reward and fear centers as well as sensory and cortical areas in ways that are associated with conscious episodic mental imagery in humans. Moreover, such hippocampal “re- and preplay” has been found to contribute to instrumental decision making, the learning of value representations, and the delay of rewards in rats. The functional and structural features of hippocampal simulation are highly conserved across mammals. This evidence makes it reasonable to assume that internally triggered experiences of hedonic valence (IHVs) are pervasive across (at least) all mammals. This conclusion has important welfare implications. Most prominently, IHVs act as a kind of “welfare multiplier” through which the welfare impacts of any given experience of pain or pleasure are increased through each future retrieval. However, IHVs also have practical implications for welfare assessment and cause prioritization.

The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning

We advance a novel computational theory of the hippocampal formation as a hierarchical generative model that organizes sequential experiences, such as rodent trajectories during spatial navigation, into coherent spatiotemporal contexts. We propose that the hippocampal generative model is endowed with inductive biases to identify individual items of experience (first hierarchical layer), organize them into sequences (second layer) and cluster them into maps (third layer). This theory entails a novel characterization of hippocampal reactivations as generative replay: the offline resampling of fictive sequences from the generative model, which supports the continual learning of multiple sequential experiences. We show that the model learns and efficiently retains multiple spatial navigation trajectories, by organizing them into spatial maps. Furthermore, the model reproduces flexible and prospective aspects of hippocampal dynamics that are challenging to explain within existing frameworks. This theory reconciles multiple roles of the hippocampal formation in map-based navigation, episodic memory and imagination.

Multiple-Timescale Representations of Space: Linking Memory to Navigation

When navigating through space, we must maintain a representation of our position in real time; when recalling a past episode, a memory can come back in a flash. Interestingly, the brain's spatial representation system, including the hippocampus, supports these two distinct timescale functions. How are neural representations of space used in the service of both real-world navigation and internal mnemonic processes? Recent progress has identified sequences of hippocampal place cells, evolving at multiple timescales in accordance with either navigational behaviors or internal oscillations, that underlie these functions. We review experimental findings on experience-dependent modulation of these sequential representations and consider how they link real-world navigation to time-compressed memories. We further discuss recent work suggesting the prevalence of these sequences beyond hippocampus and propose that these multiple-timescale mechanisms may represent a general algorithm for organizing cell assemblies, potentially unifying the dual roles of the spatial representation system in memory and navigation.

Spatial goal coding in the hippocampal formation

The mammalian hippocampal formation contains several distinct populations of neurons involved in representing self-position and orientation. These neurons, which include place, grid, head direction, and boundary-vector cells, are thought to collectively instantiate cognitive maps supporting flexible navigation. However, to flexibly navigate, it is necessary to also maintain internal representations of goal locations, such that goal-directed routes can be planned and executed. Although it has remained unclear how the mammalian brain represents goal locations, multiple neural candidates have recently been uncovered during different phases of navigation. For example, during planning, sequential activation of spatial cells may enable simulation of future routes toward the goal. During travel, modulation of spatial cells by the prospective route, or by distance and direction to the goal, may allow maintenance of route and goal-location information, supporting navigation on an ongoing basis. As the goal is approached, an increased activation of spatial cells may enable the goal location to become distinctly represented within cognitive maps, aiding goal localization. Lastly, after arrival at the goal, sequential activation of spatial cells may represent the just-taken route, enabling route learning and evaluation. Here, we review and synthesize these and other evidence for goal coding in mammalian brains, relate the experimental findings to predictions from computational models, and discuss outstanding questions and future challenges.

Hippocampal replay reflects specific past experiences rather than a plan for subsequent choice

Executing memory-guided behavior requires storage of information about experience and later recall of that information to inform choices. Awake hippocampal replay, when hippocampal neural ensembles briefly reactivate a representation related to prior experience, has been proposed to critically contribute to these memory-related processes. However, it remains unclear whether awake replay contributes to memory function by promoting the storage of past experiences, facilitating planning based on evaluation of those experiences, or both. We designed a dynamic spatial task that promotes replay before a memory-based choice and assessed how the content of replay related to past and future behavior. We found that replay content was decoupled from subsequent choice and instead was enriched for representations of previously rewarded locations and places that had not been visited recently, indicating a role in memory storage rather than in directly guiding subsequent behavior.

Hippocampal replay of experience at real-world speeds

Representations related to past experiences play a critical role in memory and decision-making processes. The rat hippocampus expresses these types of representations during sharp-wave ripple (SWR) events, and previous work identified a minority of SWRs that contain ‘replay’ of spatial trajectories at ∼20x the movement speed of the animal. Efforts to understand replay typically make multiple assumptions about which events to examine and what sorts of representations constitute replay. We therefore lack a clear understanding of both the prevalence and the range of representational dynamics associated with replay. Here, we develop a state space model that uses a combination of movement dynamics of different speeds to capture the spatial content and time evolution of replay during SWRs. Using this model, we find that the large majority of replay events contain spatially coherent, interpretable content. Furthermore, many events progress at real-world, rather than accelerated, movement speeds, consistent with actual experiences.

Learning offline: memory replay in biological and artificial reinforcement learning

Learning to act in an environment to maximise rewards is among the brain's key functions. This process has often been conceptualised within the framework of reinforcement learning, which has also gained prominence in machine learning and artificial intelligence (AI) as a way to optimise decision making. A common aspect of both biological and machine reinforcement learning is the reactivation of previously experienced episodes, referred to as replay. Replay is important for memory consolidation in biological neural networks and is key to stabilising learning in deep neural networks. Here, we review recent developments concerning the functional roles of replay in the fields of neuroscience and AI. Complementary progress suggests how replay might support learning processes, including generalisation and continual learning, affording opportunities to transfer knowledge across the two fields to advance the understanding of biological and artificial learning and memory.

Navigating for reward

An organism's survival can depend on its ability to recall and navigate to spatial locations associated with rewards, such as food or a home. Accumulating research has revealed that computations of reward and its prediction occur on multiple levels across a complex set of interacting brain regions, including those that support memory and navigation. However, how the brain coordinates the encoding, recall and use of reward information to guide navigation remains incompletely understood. In this Review, we propose that the brain's classical navigation centres - the hippocampus and the entorhinal cortex - are ideally suited to coordinate this larger network by representing both physical and mental space as a series of states. These states may be linked to reward via neuromodulatory inputs to the hippocampus-entorhinal cortex system. Hippocampal outputs can then broadcast sequences of states to the rest of the brain to store reward associations or to facilitate decision-making, potentially engaging additional value signals downstream. This proposal is supported by recent advances in both experimental and theoretical neuroscience. By discussing the neural systems traditionally tied to navigation and reward at their intersection, we aim to offer an integrated framework for understanding navigation to reward as a fundamental feature of many cognitive processes.

Distinct effects of reward and navigation history on hippocampal forward and reverse replays

To better understand the functional roles of hippocampal forward and reverse replays, we trained rats in a spatial sequence memory task and examined how these replays are modulated by reward and navigation history. We found that reward enhances both forward and reverse replays during the awake state, but in different ways. Reward enhances the rate of reverse replays, but it increases the fidelity of forward replays for recently traveled as well as other alternative trajectories heading toward a rewarding location. This suggests roles for forward and reverse replays in reinforcing representations for all potential rewarding trajectories. We also found more faithful reactivation of upcoming than already rewarded trajectories in forward replays. This suggests a role for forward replays in preferentially reinforcing representations for high-value trajectories. We propose that hippocampal forward and reverse replays might contribute to constructing a map of potential navigation trajectories and their associated values (a "value map") via distinct mechanisms.