Reactivation, replay, and preplay: how it might all fit together

Global remapping emerges as the mechanism for renewal of context-dependent behavior in a reinforcement learning model

Introduction

The hippocampal formation exhibits complex and context-dependent activity patterns and dynamics, e.g., place cell activity during spatial navigation in rodents or remapping of place fields when the animal switches between contexts. Furthermore, rodents show context-dependent renewal of extinguished behavior. However, the link between context-dependent neural codes and context-dependent renewal is not fully understood.

Methods

We use a deep neural network-based reinforcement learning agent to study the learning dynamics that occur during spatial learning and context switching in a simulated ABA extinction and renewal paradigm in a 3D virtual environment.

Results

Despite its simplicity, the network exhibits a number of features typically found in the CA1 and CA3 regions of the hippocampus. A significant proportion of neurons in deeper layers of the network are tuned to a specific spatial position of the agent in the environment-similar to place cells in the hippocampus. These complex spatial representations and dynamics occur spontaneously in the hidden layer of a deep network during learning. These spatial representations exhibit global remapping when the agent is exposed to a new context. The spatial maps are restored when the agent returns to the previous context, accompanied by renewal of the conditioned behavior. Remapping is facilitated by memory replay of experiences during training.

Discussion

Our results show that integrated codes that jointly represent spatial and task-relevant contextual variables are the mechanism underlying renewal in a simulated DQN agent.

A neural network model for online one-shot storage of pattern sequences

Based on the CRISP theory (Content Representation, Intrinsic Sequences, and Pattern completion), we present a computational model of the hippocampus that allows for online one-shot storage of pattern sequences without the need for a consolidation process. In our model, CA3 provides a pre-trained sequence that is hetero-associated with the input sequence, rather than storing a sequence in CA3. That is, plasticity on a short timescale only occurs in the incoming and outgoing connections of CA3, not in its recurrent connections. We use a single learning rule named Hebbian descent to train all plastic synapses in the network. A forgetting mechanism in the learning rule allows the network to continuously store new patterns while forgetting those stored earlier. We find that a single cue pattern can reliably trigger the retrieval of sequences, even when cues are noisy or missing information. Furthermore, pattern separation in subregion DG is necessary when sequences contain correlated patterns. Besides artificially generated input sequences, the model works with sequences of handwritten digits and natural images. Notably, our model is capable of improving itself without external input, in a process that can be referred to as 'replay' or 'offline-learning', which helps in improving the associations and consolidating the learned patterns.

Reactivation strength during cued recall is modulated by graph distance within cognitive maps

Declarative memory retrieval is thought to involve reinstatement of neuronal activity patterns elicited and encoded during a prior learning episode. Furthermore, it is suggested that two mechanisms operate during reinstatement, dependent on task demands: individual memory items can be reactivated simultaneously as a clustered occurrence or, alternatively, replayed sequentially as temporally separate instances. In the current study, participants learned associations between images that were embedded in a directed graph network and retained this information over a brief 8 min consolidation period. During a subsequent cued recall session, participants retrieved the learned information while undergoing magnetoencephalographic recording. Using a trained stimulus decoder, we found evidence for clustered reactivation of learned material. Reactivation strength of individual items during clustered reactivation decreased as a function of increasing graph distance, an ordering present solely for successful retrieval but not for retrieval failure. In line with previous research, we found evidence that sequential replay was dependent on retrieval performance and was most evident in low performers. The results provide evidence for distinct performance-dependent retrieval mechanisms, with graded clustered reactivation emerging as a plausible mechanism to search within abstract cognitive maps.

In Search of Dispersed Memories: Generative Diffusion Models Are Associative Memory Networks

Uncovering the mechanisms behind long-term memory is one of the most fascinating open problems in neuroscience and artificial intelligence. Artificial associative memory networks have been used to formalize important aspects of biological memory. Generative diffusion models are a type of generative machine learning techniques that have shown great performance in many tasks. Similar to associative memory systems, these networks define a dynamical system that converges to a set of target states. In this work, we show that generative diffusion models can be interpreted as energy-based models and that, when trained on discrete patterns, their energy function is (asymptotically) identical to that of modern Hopfield networks. This equivalence allows us to interpret the supervised training of diffusion models as a synaptic learning process that encodes the associative dynamics of a modern Hopfield network in the weight structure of a deep neural network. Leveraging this connection, we formulate a generalized framework for understanding the formation of long-term memory, where creative generation and memory recall can be seen as parts of a unified continuum.

The generative neural microdynamics of cognitive processing

The entorhinal cortex and hippocampus form a recurrent network that informs many cognitive processes, including memory, planning, navigation, and imagination. Neural recordings from these regions reveal spatially organized population codes corresponding to external environments and abstract spaces. Aligning the former cognitive functionalities with the latter neural phenomena is a central challenge in understanding the entorhinal-hippocampal circuit (EHC). Disparate experiments demonstrate a surprising level of complexity and apparent disorder in the intricate spatiotemporal dynamics of sequential non-local hippocampal reactivations, which occur particularly, though not exclusively, during immobile pauses and rest. We review these phenomena with a particular focus on their apparent lack of physical simulative realism. These observations are then integrated within a theoretical framework and proposed neural circuit mechanisms that normatively characterize this neural complexity by conceiving different regimes of hippocampal microdynamics as neuromarkers of diverse cognitive computations.

Altered patterning of neural activity in a tauopathy mouse model

Alzheimer's disease (AD) is a complex neurodegenerative condition that manifests at multiple levels and involves a spectrum of abnormalities ranging from the cellular to cognitive. Here, we investigate the impact of AD-related tau-pathology on hippocampal circuits in mice engaged in spatial navigation, and study changes of neuronal firing and dynamics of extracellular fields. While most studies are based on analyzing instantaneous or time-averaged characteristics of neuronal activity, we focus on intermediate timescales-spike trains and waveforms of oscillatory potentials, which we consider as single entities. We find that, in healthy mice, spike arrangements and wave patterns (series of crests or troughs) are coupled to the animal's location, speed, and acceleration. In contrast, in tau-mice, neural activity is structurally disarrayed: brainwave cadence is detached from locomotion, spatial selectivity is lost, the spike flow is scrambled. Importantly, these alterations start early and accumulate with age, which exposes progressive disinvolvement the hippocampus circuit in spatial navigation. These features highlight qualitatively different neurodynamics than the ones provided by conventional analyses, and are more salient, thus revealing a new level of the hippocampal circuit disruptions.

A theory of the neural mechanisms underlying negative cognitive bias in major depression

The widely acknowledged cognitive theory of depression, developed by Aaron Beck, focused on biased information processing that emphasizes the negative aspects of affective and conceptual information. Current attempts to discover the neurological mechanism underlying such cognitive and affective bias have successfully identified various brain regions associated with severally biased functions such as emotion, attention, rumination, and inhibition control. However, the neurobiological mechanisms of how individuals in depression develop this selective processing toward negative is still under question. This paper introduces a neurological framework centered around the frontal-limbic circuit, specifically analyzing and synthesizing the activity and functional connectivity within the amygdala, hippocampus, and medial prefrontal cortex. Firstly, a possible explanation of how the positive feedback loop contributes to the persistent hyperactivity of the amygdala in depression at an automatic level is established. Building upon this, two hypotheses are presented: hypothesis 1 revolves around the bidirectional amygdalohippocampal projection facilitating the amplification of negative emotions and memories while concurrently contributing to the impediment of the retrieval of opposing information in the hippocampus attractor network. Hypothesis 2 highlights the involvement of the ventromedial prefrontal cortex in the establishment of a negative cognitive framework through the generalization of conceptual and emotional information in conjunction with the amygdala and hippocampus. The primary objective of this study is to improve and complement existing pathological models of depression, pushing the frontiers of current understanding in neuroscience of affective disorders, and eventually contributing to successful recovery from the debilitating affective disorders.

A theory of hippocampal theta correlations accounting for extrinsic and intrinsic sequences

Hippocampal place cell sequences have been hypothesized to serve as diverse purposes as the induction of synaptic plasticity, formation and consolidation of long-term memories, or navigation and planning. During spatial behaviors of rodents, sequential firing of place cells at the theta timescale (known as theta sequences) encodes running trajectories, which can be considered as one-dimensional behavioral sequences of traversed locations. In a two-dimensional space, however, each single location can be visited along arbitrary one-dimensional running trajectories. Thus, a place cell will generally take part in multiple different theta sequences, raising questions about how this two-dimensional topology can be reconciled with the idea of hippocampal sequences underlying memory of (one-dimensional) episodes. Here, we propose a computational model of cornu ammonis 3 (CA3) and dentate gyrus (DG), where sensorimotor input drives the direction-dependent (extrinsic) theta sequences within CA3 reflecting the two-dimensional spatial topology, whereas the intrahippocampal CA3-DG projections concurrently produce intrinsic sequences that are independent of the specific running trajectory. Consistent with experimental data, intrinsic theta sequences are less prominent, but can nevertheless be detected during theta activity, thereby serving as running-direction independent landmark cues. We hypothesize that the intrinsic sequences largely reflect replay and preplay activity during non-theta states.

Rapid runtime learning by curating small datasets of high-quality items obtained from memory

We propose the "runtime learning" hypothesis which states that people quickly learn to perform unfamiliar tasks as the tasks arise by using task-relevant instances of concepts stored in memory during mental training. To make learning rapid, the hypothesis claims that only a few class instances are used, but these instances are especially valuable for training. The paper motivates the hypothesis by describing related ideas from the cognitive science and machine learning literatures. Using computer simulation, we show that deep neural networks (DNNs) can learn effectively from small, curated training sets, and that valuable training items tend to lie toward the centers of data item clusters in an abstract feature space. In a series of three behavioral experiments, we show that people can also learn effectively from small, curated training sets. Critically, we find that participant reaction times and fitted drift rates are best accounted for by the confidences of DNNs trained on small datasets of highly valuable items. We conclude that the runtime learning hypothesis is a novel conjecture about the relationship between learning and memory with the potential for explaining a wide variety of cognitive phenomena.

A map of spatial navigation for neuroscience

Spatial navigation has received much attention from neuroscientists, leading to the identification of key brain areas and the discovery of numerous spatially selective cells. Despite this progress, our understanding of how the pieces fit together to drive behavior is generally lacking. We argue that this is partly caused by insufficient communication between behavioral and neuroscientific researchers. This has led the latter to under-appreciate the relevance and complexity of spatial behavior, and to focus too narrowly on characterizing neural representations of space-disconnected from the computations these representations are meant to enable. We therefore propose a taxonomy of navigation processes in mammals that can serve as a common framework for structuring and facilitating interdisciplinary research in the field. Using the taxonomy as a guide, we review behavioral and neural studies of spatial navigation. In doing so, we validate the taxonomy and showcase its usefulness in identifying potential issues with common experimental approaches, designing experiments that adequately target particular behaviors, correctly interpreting neural activity, and pointing to new avenues of research.

Representational drift as a window into neural and behavioural plasticity

Large-scale recordings of neural activity over days and weeks have revealed that neural representations of familiar tasks, precepts and actions continually evolve without obvious changes in behaviour. We hypothesise that this steady drift in neural activity and accompanying physiological changes is due in part to the continuous application of a learning rule at the cellular and population level. Explicit predictions of this drift can be found in neural network models that use iterative learning to optimise weights. Drift therefore provides a measurable signal that can reveal systems-level properties of biological plasticity mechanisms, such as their precision and effective learning rates.

Modeling the function of episodic memory in spatial learning

Episodic memory has been studied extensively in the past few decades, but so far little is understood about how it drives future behavior. Here we propose that episodic memory can facilitate learning in two fundamentally different modes: retrieval and replay, which is the reinstatement of hippocampal activity patterns during later sleep or awake quiescence. We study their properties by comparing three learning paradigms using computational modeling based on visually-driven reinforcement learning. Firstly, episodic memories are retrieved to learn from single experiences (one-shot learning); secondly, episodic memories are replayed to facilitate learning of statistical regularities (replay learning); and, thirdly, learning occurs online as experiences arise with no access to memories of past experiences (online learning). We found that episodic memory benefits spatial learning in a broad range of conditions, but the performance difference is meaningful only when the task is sufficiently complex and the number of learning trials is limited. Furthermore, the two modes of accessing episodic memory affect spatial learning differently. One-shot learning is typically faster than replay learning, but the latter may reach a better asymptotic performance. In the end, we also investigated the benefits of sequential replay and found that replaying stochastic sequences results in faster learning as compared to random replay when the number of replays is limited. Understanding how episodic memory drives future behavior is an important step toward elucidating the nature of episodic memory.

Bayesian continual learning via spiking neural networks

Among the main features of biological intelligence are energy efficiency, capacity for continual adaptation, and risk management via uncertainty quantification. Neuromorphic engineering has been thus far mostly driven by the goal of implementing energy-efficient machines that take inspiration from the time-based computing paradigm of biological brains. In this paper, we take steps toward the design of neuromorphic systems that are capable of adaptation to changing learning tasks, while producing well-calibrated uncertainty quantification estimates. To this end, we derive online learning rules for spiking neural networks (SNNs) within a Bayesian continual learning framework. In it, each synaptic weight is represented by parameters that quantify the current epistemic uncertainty resulting from prior knowledge and observed data. The proposed online rules update the distribution parameters in a streaming fashion as data are observed. We instantiate the proposed approach for both real-valued and binary synaptic weights. Experimental results using Intel's Lava platform show the merits of Bayesian over frequentist learning in terms of capacity for adaptation and uncertainty quantification.

Freezing revisited: coordinated autonomic and central optimization of threat coping

Animals have sophisticated mechanisms for coping with danger. Freezing is a unique state that, upon threat detection, allows evidence to be gathered, response possibilities to be previsioned and preparations to be made for worst-case fight or flight. We propose that - rather than reflecting a passive fear state - the particular somatic and cognitive characteristics of freezing help to conceal overt responses, while optimizing sensory processing and action preparation. Critical for these functions are the neurotransmitters noradrenaline and acetylcholine, which modulate neural information processing and also control the sympathetic and parasympathetic branches of the autonomic nervous system. However, the interactions between autonomic systems and the brain during freezing, and the way in which they jointly coordinate responses, remain incompletely explored. We review the joint actions of these systems and offer a novel computational framework to describe their temporally harmonized integration. This reconceptualization of freezing has implications for its role in decision-making under threat and for psychopathology.

Eye movements reveal spatiotemporal dynamics of visually-informed planning in navigation

Goal-oriented navigation is widely understood to depend upon internal maps. Although this may be the case in many settings, humans tend to rely on vision in complex, unfamiliar environments. To study the nature of gaze during visually-guided navigation, we tasked humans to navigate to transiently visible goals in virtual mazes of varying levels of difficulty, observing that they took near-optimal trajectories in all arenas. By analyzing participants' eye movements, we gained insights into how they performed visually-informed planning. The spatial distribution of gaze revealed that environmental complexity mediated a striking trade-off in the extent to which attention was directed towards two complimentary aspects of the world model: the reward location and task-relevant transitions. The temporal evolution of gaze revealed rapid, sequential prospection of the future path, evocative of neural replay. These findings suggest that the spatiotemporal characteristics of gaze during navigation are significantly shaped by the unique cognitive computations underlying real-world, sequential decision making.

Neural Circuits for Dynamics-Based Segmentation of Time Series

The brain must extract behaviorally relevant latent variables from the signals streamed by the sensory organs. Such latent variables are often encoded in the dynamics that generated the signal rather than in the specific realization of the waveform. Therefore, one problem faced by the brain is to segment time series based on underlying dynamics. We present two algorithms for performing this segmentation task that are biologically plausible, which we define as acting in a streaming setting and all learning rules being local. One algorithm is model based and can be derived from an optimization problem involving a mixture of autoregressive processes. This algorithm relies on feedback in the form of a prediction error and can also be used for forecasting future samples. In some brain regions, such as the retina, the feedback connections necessary to use the prediction error for learning are absent. For this case, we propose a second, model-free algorithm that uses a running estimate of the autocorrelation structure of the signal to perform the segmentation. We show that both algorithms do well when tasked with segmenting signals drawn from autoregressive models with piecewise-constant parameters. In particular, the segmentation accuracy is similar to that obtained from oracle-like methods in which the ground-truth parameters of the autoregressive models are known. We also test our methods on data sets generated by alternating snippets of voice recordings. We provide implementations of our algorithms at https://github.com/ttesileanu/bio-time-series.

Replay in minds and machines

Experience-related brain activity patterns reactivate during sleep, wakeful rest, and brief pauses from active behavior. In parallel, machine learning research has found that experience replay can lead to substantial performance improvements in artificial agents. Together, these lines of research suggest replay has a variety of computational benefits for decision-making and learning. Here, we provide an overview of putative computational functions of replay as suggested by machine learning and neuroscientific research. We show that replay can lead to faster learning, less forgetting, reorganization or augmentation of experiences, and support planning and generalization. In addition, we highlight the benefits of reactivating abstracted internal representations rather than veridical memories, and discuss how replay could provide a mechanism to build internal representations that improve learning and decision-making.

Rest Is Required to Learn an Appetitively-Reinforced Operant Task in Drosophila

Maladaptive operant conditioning contributes to development of neuropsychiatric disorders. Candidate genes have been identified that contribute to this maladaptive plasticity, but the neural basis of operant conditioning in genetic model organisms remains poorly understood. The fruit fly Drosophila melanogaster is a versatile genetic model organism that readily forms operant associations with punishment stimuli. However, operant conditioning with a food reward has not been demonstrated in flies, limiting the types of neural circuits that can be studied. Here we present the first sucrose-reinforced operant conditioning paradigm for flies. In the paradigm, flies walk along a Y-shaped track with reward locations at the terminus of each hallway. When flies turn in the reinforced direction at the center of the track, they receive a sucrose reward at the end of the hallway. Only flies that rest early in training learn the reward contingency normally. Flies rewarded independently of their behavior do not form a learned association but have the same amount of rest as trained flies, showing that rest is not driven by learning. Optogenetically-induced sleep does not promote learning, indicating that sleep itself is not sufficient for learning the operant task. We validated the sensitivity of this assay to detect the effect of genetic manipulations by testing the classic learning mutant dunce. Dunce flies are learning-impaired in the Y-Track task, indicating a likely role for cAMP in the operant coincidence detector. This novel training paradigm will provide valuable insight into the molecular mechanisms of disease and the link between sleep and learning.

Context-dependent extinction learning emerging from raw sensory inputs: a reinforcement learning approach

The context-dependence of extinction learning has been well studied and requires the hippocampus. However, the underlying neural mechanisms are still poorly understood. Using memory-driven reinforcement learning and deep neural networks, we developed a model that learns to navigate autonomously in biologically realistic virtual reality environments based on raw camera inputs alone. Neither is context represented explicitly in our model, nor is context change signaled. We find that memory-intact agents learn distinct context representations, and develop ABA renewal, whereas memory-impaired agents do not. These findings reproduce the behavior of control and hippocampal animals, respectively. We therefore propose that the role of the hippocampus in the context-dependence of extinction learning might stem from its function in episodic-like memory and not in context-representation per se. We conclude that context-dependence can emerge from raw visual inputs.

Dark control: The default mode network as a reinforcement learning agent

The default mode network (DMN) is believed to subserve the baseline mental activity in humans. Its higher energy consumption compared to other brain networks and its intimate coupling with conscious awareness are both pointing to an unknown overarching function. Many research streams speak in favor of an evolutionarily adaptive role in envisioning experience to anticipate the future. In the present work, we propose a process model that tries to explain how the DMN may implement continuous evaluation and prediction of the environment to guide behavior. The main purpose of DMN activity, we argue, may be described by Markov decision processes that optimize action policies via value estimates through vicarious trial and error. Our formal perspective on DMN function naturally accommodates as special cases previous interpretations based on (a) predictive coding, (b) semantic associations, and (c) a sentinel role. Moreover, this process model for the neural optimization of complex behavior in the DMN offers parsimonious explanations for recent experimental findings in animals and humans.

Low acetylcholine during early sleep is important for motor memory consolidation

The synaptic homeostasis theory of sleep proposes that low neurotransmitter activity in sleep optimizes memory consolidation. We tested this theory by asking whether increasing acetylcholine levels during early sleep would weaken motor memory consolidation. We trained separate groups of adult mice on the rotarod walking task and the single pellet reaching task, and after training, administered physostigmine, an acetylcholinesterase inhibitor, to increase cholinergic tone in subsequent sleep. Post-sleep testing showed that physostigmine impaired motor skill acquisition of both tasks. Home-cage video monitoring and electrophysiology revealed that physostigmine disrupted sleep structure, delayed non-rapid-eye-movement sleep onset, and reduced slow-wave power in the hippocampus and cortex. Additional experiments showed that: (1) the impaired performance associated with physostigmine was not due to its effects on sleep structure, as 1 h of sleep deprivation after training did not impair rotarod performance, (2) a reduction in cholinergic tone by inactivation of cholinergic neurons during early sleep did not affect rotarod performance, and (3) stimulating or blocking muscarinic and nicotinic acetylcholine receptors did not impair rotarod performance. Taken together, the experiments suggest that the increased slow wave activity and inactivation of both muscarinic and nicotinic receptors during early sleep due to reduced acetylcholine contribute to motor memory consolidation.

Sleep and synaptic down-selection

The synaptic homeostasis hypothesis (SHY) proposes that sleep is an essential process needed by the brain to maintain the total amount of synaptic strength under control. SHY predicts that by the end of a waking day the synaptic connections of many neural circuits undergo a net increase in synaptic strength due to ongoing learning, which is mainly mediated by synaptic potentiation. Stronger synapses require more energy and supplies and are prone to saturation, creating the need for synaptic renormalization. Such renormalization should mainly occur during sleep, when the brain is disconnected from the environment and neural circuits can be broadly reactivated off-line to undergo a systematic but specific synaptic down-selection. In short, according to SHY sleep is the price to pay for waking plasticity, to avoid runaway potentiation, decreased signal-to-noise ratio, and impaired learning due to saturation. In this review, we briefly discuss the rationale of the hypothesis and recent supportive ultrastructural evidence obtained in our laboratory. We then examine recent studies by other groups showing the causal role of cortical slow waves and hippocampal sharp waves/ripples in sleep-dependent down-selection of neural activity and synaptic strength. Finally, we discuss some of the molecular mechanisms that could mediate synaptic weakening during sleep.

Brain Rhythms During Sleep and Memory Consolidation: Neurobiological Insights

Sleep can benefit memory consolidation. The characterization of brain regions underlying memory consolidation during sleep, as well as their temporal interplay, reflected by specific patterns of brain electric activity, is surfacing. Here, we provide an overview of recent concepts and results on the mechanisms of sleep-related memory consolidation. The latest studies strongly impacting future directions of research in this field are highlighted.

Hippocampal Sequences During Exploration: Mechanisms and Functions

Although the hippocampus plays a critical role in spatial and episodic memories, the mechanisms underlying memory formation, stabilization, and recall for adaptive behavior remain relatively unknown. During exploration, within single cycles of the ongoing theta rhythm that dominates hippocampal local field potentials, place cells form precisely ordered sequences of activity. These neural sequences result from the integration of both external inputs conveying sensory-motor information, and intrinsic network dynamics possibly related to memory processes. Their endogenous replay during subsequent sleep is critical for memory consolidation. The present review discusses possible mechanisms and functions of hippocampal theta sequences during exploration. We present several lines of evidence suggesting that these neural sequences play a key role in information processing and support the formation of initial memory traces, and discuss potential functional distinctions between neural sequences emerging during theta vs. awake sharp-wave ripples.

NEVER forget: negative emotional valence enhances recapitulation

A hallmark feature of episodic memory is that of "mental time travel," whereby an individual feels they have returned to a prior moment in time. Cognitive and behavioral neuroscience methods have revealed a neurobiological counterpart: Successful retrieval often is associated with reactivation of a prior brain state. We review the emerging literature on memory reactivation and recapitulation, and we describe evidence for the effects of emotion on these processes. Based on this review, we propose a new model: Negative Emotional Valence Enhances Recapitulation (NEVER). This model diverges from existing models of emotional memory in three key ways. First, it underscores the effects of emotion during retrieval. Second, it stresses the importance of sensory processing to emotional memory. Third, it emphasizes how emotional valence - whether an event is negative or positive - affects the way that information is remembered. The model specifically proposes that, as compared to positive events, negative events both trigger increased encoding of sensory detail and elicit a closer resemblance between the sensory encoding signature and the sensory retrieval signature. The model also proposes that negative valence enhances the reactivation and storage of sensory details over offline periods, leading to a greater divergence between the sensory recapitulation of negative and positive memories over time. Importantly, the model proposes that these valence-based differences occur even when events are equated for arousal, thus rendering an exclusively arousal-based theory of emotional memory insufficient. We conclude by discussing implications of the model and suggesting directions for future research to test the tenets of the model.

Overlap between hippocampal pre-encoding and encoding patterns supports episodic memory

It is well-established that whether the information will be remembered or not depends on the extent to which the learning context is reinstated during post-encoding rest and/or at retrieval. It has yet to be determined, however, if the fundamental importance of contextual reinstatement to memory extends to periods of spontaneous neurocognitive activity prior to learning. We thus asked whether memory performance can be predicted by the extent to which spontaneous pre-encoding neural patterns resemble patterns elicited during encoding. Individuals studied and retrieved lists of words while undergoing fMRI-scanning. Multivoxel hippocampal patterns during resting periods prior to encoding resembled hippocampal patterns at encoding most strongly for items that were subsequently remembered. Furthermore, across subjects, the magnitude of similarity correlated with a behavioral measure of episodic recall. The results indicate that the neural context before learning is an important determinant of memory.

Prioritized memory access explains planning and hippocampal replay

To make decisions, animals must evaluate candidate choices by accessing memories of relevant experiences. Yet little is known about which experiences are considered or ignored during deliberation, which ultimately governs choice. We propose a normative theory predicting which memories should be accessed at each moment to optimize future decisions. Using nonlocal 'replay' of spatial locations in hippocampus as a window into memory access, we simulate a spatial navigation task in which an agent accesses memories of locations sequentially, ordered by utility: how much extra reward would be earned due to better choices. This prioritization balances two desiderata: the need to evaluate imminent choices versus the gain from propagating newly encountered information to preceding locations. Our theory offers a simple explanation for numerous findings about place cells; unifies seemingly disparate proposed functions of replay including planning, learning, and consolidation; and posits a mechanism whose dysfunction may underlie pathologies like rumination and craving.

Experience Replay Using Transition Sequences

Experience replay is one of the most commonly used approaches to improve the sample efficiency of reinforcement learning algorithms. In this work, we propose an approach to select and replay sequences of transitions in order to accelerate the learning of a reinforcement learning agent in an off-policy setting. In addition to selecting appropriate sequences, we also artificially construct transition sequences using information gathered from previous agent-environment interactions. These sequences, when replayed, allow value function information to trickle down to larger sections of the state/state-action space, thereby making the most of the agent's experience. We demonstrate our approach on modified versions of standard reinforcement learning tasks such as the mountain car and puddle world problems and empirically show that it enables faster, and more accurate learning of value functions as compared to other forms of experience replay. Further, we briefly discuss some of the possible extensions to this work, as well as applications and situations where this approach could be particularly useful.

Uncovering temporal structure in hippocampal output patterns

Place cell activity of hippocampal pyramidal cells has been described as the cognitive substrate of spatial memory. Replay is observed during hippocampal sharp-wave-ripple-associated population burst events (PBEs) and is critical for consolidation and recall-guided behaviors. PBE activity has historically been analyzed as a phenomenon subordinate to the place code. Here, we use hidden Markov models to study PBEs observed in rats during exploration of both linear mazes and open fields. We demonstrate that estimated models are consistent with a spatial map of the environment, and can even decode animals’ positions during behavior. Moreover, we demonstrate the model can be used to identify hippocampal replay without recourse to the place code, using only PBE model congruence. These results suggest that downstream regions may rely on PBEs to provide a substrate for memory. Additionally, by forming models independent of animal behavior, we lay the groundwork for studies of non-spatial memory.

Theta Neurofeedback Effects on Motor Memory Consolidation and Performance Accuracy: An Apparent Paradox?

Previous studies have shown that theta neurofeedback enhances motor memory consolidation on an easy-to-learn finger-tapping task. However, the simplicity of the finger-tapping task precludes evaluating the putative effects of elevated theta on performance accuracy. Mastering a motor sequence is classically assumed to entail faster performance with fewer errors. The speed-accuracy tradeoff (SAT) principle states that as action speed increases, motor performance accuracy decreases. The current study investigated whether theta neurofeedback could improve both performance speed and performance accuracy, or would only enhance performance speed at the cost of reduced accuracy. A more complex task was used to study the effects of parietal elevated theta on 45 healthy volunteers The findings confirmed previous results on the effects of theta neurofeedback on memory consolidation. In contrast to the two control groups, in the theta-neurofeedback group the speed-accuracy tradeoff was reversed. The speed-accuracy tradeoff patterns only stabilized after a night's sleep implying enhancement in terms of both speed and accuracy.

Resonance with subthreshold oscillatory drive organizes activity and optimizes learning in neural networks

Networks of neurons need to reliably encode and replay patterns and sequences of activity. In the brain, sequences of spatially coding neurons are replayed in both the forward and reverse direction in time with respect to their order in recent experience. As of yet there is no network-level or biophysical mechanism known that can produce both modes of replay within the same network. Here we propose that resonance, a property of neurons, paired with subthreshold oscillations in neural input facilitate network-level learning of fixed and sequential activity patterns and lead to both forward and reverse replay.

Network oscillations across and within brain areas are critical for learning and performance of memory tasks. While a large amount of work has focused on the generation of neural oscillations, their effect on neuronal populations’ spiking activity and information encoding is less known. Here, we use computational modeling to demonstrate that a shift in resonance responses can interact with oscillating input to ensure that networks of neurons properly encode new information represented in external inputs to the weights of recurrent synaptic connections. Using a neuronal network model, we find that due to an input current-dependent shift in their resonance response, individual neurons in a network will arrange their phases of firing to represent varying strengths of their respective inputs. As networks encode information, neurons fire more synchronously, and this effect limits the extent to which further “learning” (in the form of changes in synaptic strength) can occur. We also demonstrate that sequential patterns of neuronal firing can be accurately stored in the network; these sequences are later reproduced without external input (in the context of subthreshold oscillations) in both the forward and reverse directions (as has been observed following learning in vivo). To test whether a similar mechanism could act in vivo, we show that periodic stimulation of hippocampal neurons coordinates network activity and functional connectivity in a frequency-dependent manner. We conclude that resonance with subthreshold oscillations provides a plausible network-level mechanism to accurately encode and retrieve information without overstrengthening connections between neurons.

The Interaction between Semantic Representation and Episodic Memory

The experimental evidence on the interrelation between episodic memory and semantic memory is inconclusive. Are they independent systems, different aspects of a single system, or separate but strongly interacting systems? Here, we propose a computational role for the interaction between the semantic and episodic systems that might help resolve this debate. We hypothesize that episodic memories are represented as sequences of activation patterns. These patterns are the output of a semantic representational network that compresses the high-dimensional sensory input. We show quantitatively that the accuracy of episodic memory crucially depends on the quality of the semantic representation. We compare two types of semantic representations: appropriate representations, which means that the representation is used to store input sequences that are of the same type as those that it was trained on, and inappropriate representations, which means that stored inputs differ from the training data. Retrieval accuracy is higher for appropriate representations because the encoded sequences are less divergent than those encoded with inappropriate representations. Consistent with our model prediction, we found that human subjects remember some aspects of episodes significantly more accurately if they had previously been familiarized with the objects occurring in the episode, as compared to episodes involving unfamiliar objects. We thus conclude that the interaction with the semantic system plays an important role for episodic memory.

Internally generated hippocampal sequences as a vantage point to probe future-oriented cognition

Information processing in the rodent hippocampus is fundamentally shaped by internally generated sequences (IGSs), expressed during two different network states: theta sequences, which repeat and reset at the ∼8 Hz theta rhythm associated with active behavior, and punctate sharp wave-ripple (SWR) sequences associated with wakeful rest or slow-wave sleep. A potpourri of diverse functional roles has been proposed for these IGSs, resulting in a fragmented conceptual landscape. Here, we advance a unitary view of IGSs, proposing that they reflect an inferential process that samples a policy from the animal's generative model, supported by hippocampus-specific priors. The same inference affords different cognitive functions when the animal is in distinct dynamical modes, associated with specific functional networks. Theta sequences arise when inference is coupled to the animal's action-perception cycle, supporting online spatial decisions, predictive processing, and episode encoding. SWR sequences arise when the animal is decoupled from the action-perception cycle and may support offline cognitive processing, such as memory consolidation, the prospective simulation of spatial trajectories, and imagination. We discuss the empirical bases of this proposal in relation to rodent studies and highlight how the proposed computational principles can shed light on the mechanisms of future-oriented cognition in humans.

Sleep does not facilitate insight in older adults

Sleep has been shown to foster the process of insight generation in young adults during problem solving activities. Aging is characterized by substantial changes in sleep architecture altering memory consolidation. Whether sleep might promote the occurrence of insight in older adults as well has not yet been tested experimentally. To address this issue, we tested healthy young and old volunteers on an insight problem solving task, involving both explicit and implicit features, before and after a night of sleep or a comparable wakefulness period. Data showed that insight emerged significantly less frequently after a night of sleep in older adults compared to young. Moreover, there was no difference in the magnitude of insight occurrence following sleep and daytime -consolidation in aged participants. We further found that acquisition of implicit knowledge in the task before sleep potentiated the gain of insight in young participants, but this effect was not observed in aged participants. Overall, present findings demonstrate that a period of sleep does not significantly promote insight in problem solving in older adults.

Consolidation of Prospective Memory: Effects of Sleep on Completed and Reinstated Intentions

Sleep has been shown to facilitate the consolidation of prospective memory, which is the ability to execute intended actions at the appropriate time in the future. In a previous study, the sleep benefit for prospective memory was mainly expressed as a preservation of prospective memory performance under divided attention as compared to full attention. Based on evidence that intentions are only remembered as long as they have not been executed yet (cf. 'Zeigarnik effect'), here we asked whether the enhancement of prospective memory by sleep vanishes if the intention is completed before sleep and whether completed intentions can be reinstated to benefit from sleep again. In Experiment 1, subjects learned cue-associate word pairs in the evening and were prospectively instructed to detect the cue words and to type in the associates in a lexical decision task (serving as ongoing task) 2 h later before a night of sleep or wakefulness. At a second surprise test 2 days later, sleep and wake subjects did not differ in prospective memory performance. Specifically, both sleep and wake groups detected fewer cue words under divided compared to full attention, indicating that sleep does not facilitate the consolidation of completed intentions. Unexpectedly, in Experiment 2, reinstating the intention, by instructing subjects about the second test after completion of the first test, was not sufficient to restore the sleep benefit. However, in Experiment 3, where subjects were instructed about both test sessions immediately after learning, sleep facilitated prospective memory performance at the second test after 2 days, evidenced by comparable cue word detection under divided attention and full attention in sleep participants, whereas wake participants detected fewer cue words under divided relative to full attention. Together, these findings show that for prospective memory to benefit from sleep, (i) the intention has to be active across the sleep period, and (ii) the intention should be induced in temporal proximity to the initial learning session.

A hidden Markov model for decoding and the analysis of replay in spike trains

We present a hidden Markov model that describes variation in an animal's position associated with varying levels of activity in action potential spike trains of individual place cell neurons. The model incorporates a coarse-graining of position, which we find to be a more parsimonious description of the system than other models. We use a sequential Monte Carlo algorithm for Bayesian inference of model parameters, including the state space dimension, and we explain how to estimate position from spike train observations (decoding). We obtain greater accuracy over other methods in the conditions of high temporal resolution and small neuronal sample size. We also present a novel, model-based approach to the study of replay: the expression of spike train activity related to behaviour during times of motionlessness or sleep, thought to be integral to the consolidation of long-term memories. We demonstrate how we can detect the time, information content and compression rate of replay events in simulated and real hippocampal data recorded from rats in two different environments, and verify the correlation between the times of detected replay events and of sharp wave/ripples in the local field potential.

Sleep as spatiotemporal integration of biological processes that evolved to periodically reinforce neurodynamic and metabolic homeostasis: The 2m3d paradigm of sleep

Sleep continues to perplex scientists and researchers. Despite decades of sleep research, we still lack a clear understanding of the biological functions and evolution of sleep. In this review, we will examine sleep from a functional and phylogenetic perspective and describe some important conceptual gaps in understanding sleep. Classical theories of the biology and evolution of sleep emphasize sensory activation, energy balance, and metabolic homeostasis. Advances in electrophysiology, functional neuroimaging, and neuroplasticity allow us to view sleep within the framework of neural dynamics. With this paradigm shift, we have come to realize the importance of neurodynamic homeostasis in shaping the biology of sleep. Evidently, animals sleep to achieve neurodynamic and metabolic homeostasis. We are not aware of any framework for understanding sleep where neurodynamic, metabolic, homeostatic, chronophasic, and afferent variables are all taken into account. This motivated us to propose the two-mode three-drive (2m3d) paradigm of sleep. In the 2m3d paradigm, local neurodynamic/metabolic (N/M) processes switch between two modes-m0 and m1-in response to three drives-afferent, chronophasic, and homeostatic. The spatiotemporal integration of local m0/m1 operations gives rise to the global states of sleep and wakefulness. As a framework of evolution, the 2m3d paradigm allows us to view sleep as a robust adaptive strategy that evolved so animals can periodically reinforce neurodynamic and metabolic homeostasis while remaining sensitive to their internal and external environment.

Topological Schemas of Cognitive Maps and Spatial Learning

Spatial navigation in mammals is based on building a mental representation of their environment-a cognitive map. However, both the nature of this cognitive map and its underpinning in neural structures and activity remains vague. A key difficulty is that these maps are collective, emergent phenomena that cannot be reduced to a simple combination of inputs provided by individual neurons. In this paper we suggest computational frameworks for integrating the spiking signals of individual cells into a spatial map, which we call schemas. We provide examples of four schemas defined by different types of topological relations that may be neurophysiologically encoded in the brain and demonstrate that each schema provides its own large-scale characteristics of the environment-the schema integrals. Moreover, we find that, in all cases, these integrals are learned at a rate which is faster than the rate of complete training of neural networks. Thus, the proposed schema framework differentiates between the cognitive aspect of spatial learning and the physiological aspect at the neural network level.

Short-Range Temporal Interactions in Sleep; Hippocampal Spike Avalanches Support a Large Milieu of Sequential Activity Including Replay

Hippocampal neural systems consolidate multiple complex behaviors into memory. However, the temporal structure of neural firing supporting complex memory consolidation is unknown. Replay of hippocampal place cells during sleep supports the view that a simple repetitive behavior modifies sleep firing dynamics, but does not explain how multiple episodes could be integrated into associative networks for recollection during future cognition. Here we decode sequential firing structure within spike avalanches of all pyramidal cells recorded in sleeping rats after running in a circular track. We find that short sequences that combine into multiple long sequences capture the majority of the sequential structure during sleep, including replay of hippocampal place cells. The ensemble, however, is not optimized for maximally producing the behavior-enriched episode. Thus behavioral programming of sequential correlations occurs at the level of short-range interactions, not whole behavioral sequences and these short sequences are assembled into a large and complex milieu that could support complex memory consolidation.

Dissociating memory traces and scenario construction in mental time travel

There has been a persistent debate about how to define episodic memory and whether it is a uniquely human capacity. On the one hand, many animal cognition studies employ content-based criteria, such as the what-where-when criterion, and argue that nonhuman animals possess episodic memory. On the other hand, many human cognition studies emphasize the subjective experience during retrieval as an essential property of episodic memory and the distinctly human foresight it purportedly enables. We propose that both perspectives may examine distinct but complementary aspects of episodic memory by drawing a conceptual distinction between episodic memory traces and mental time travel. Episodic memory traces are sequential mnemonic representations of particular, personally experienced episodes. Mental time travel draws on these traces, but requires other components to construct scenarios and embed them into larger narratives. Various nonhuman animals may store episodic memory traces, and yet it is possible that only humans are able to construct and reflect on narratives of their lives - and flexibly compare alternative scenarios of the remote future.

Is there a pilot in the brain? Contribution of the self-positioning system to spatial navigation

Since the discovery of place cells, the hippocampus is thought to be the neural substrate of a cognitive map. The later discovery of head direction cells, grid cells and border cells, as well as of cells with more complex spatial signals, has led to the idea that there is a brain system devoted to providing the animal with the information required to achieve efficient navigation. Current questioning is focused on how these signals are integrated in the brain. In this review, we focus on the issue of how self-localization is performed in the hippocampal place cell map. To do so, we first shortly review the sensory information used by place cells and then explain how this sensory information can lead to two coding modes, respectively based on external landmarks (allothetic information) and self-motion cues (idiothetic information). We hypothesize that these two modes can be used concomitantly with the rat shifting from one mode to the other during its spatial displacements. We then speculate that sequential reactivation of place cells could participate in the resetting of self-localization under specific circumstances and in learning a new environment. Finally, we provide some predictions aimed at testing specific aspects of the proposed ideas.

Hippocampal place cells construct reward related sequences through unexplored space

Dominant theories of hippocampal function propose that place cell representations are formed during an animal's first encounter with a novel environment and are subsequently replayed during off-line states to support consolidation and future behaviour. Here we report that viewing the delivery of food to an unvisited portion of an environment leads to off-line pre-activation of place cells sequences corresponding to that space. Such ‘preplay’ was not observed for an unrewarded but otherwise similar portion of the environment. These results suggest that a hippocampal representation of a visible, yet unexplored environment can be formed if the environment is of motivational relevance to the animal. We hypothesise such goal-biased preplay may support preparation for future experiences in novel environments.

DOI:http://dx.doi.org/10.7554/eLife.06063.001

As an animal explores an area, part of the brain called the hippocampus creates a mental map of the space. When the animal is in one location, a few neurons called ‘place cells’ will fire. If the animal moves to a new spot, other place cells fire instead. Each time the animal returns to that spot, the same place cells will fire. Thus, as the animal moves, a place-specific pattern of firing emerges that scientists can view by recording the cells' activity and which can be used to reconstruct the animal's position.

After exploring a space, the hippocampus may replay the new place-specific pattern of activity during sleep. By doing so, the brain consolidates the memory of the space for return visits. Recent evidence now suggests that these mental rehearsals—or internal simulations of the space—may begin even before a new space has been explored.

Now, Ólafsdóttir, Barry et al. report that whether an animal's brain simulates a first visit to a new space depends on whether the animal anticipates a reward. In the experiments, rats were allowed to run up to the junction in a T-shaped track. The animals could see into each of the arms, but not enter them. Food was then placed in one of the inaccessible arms. Ólafsdóttir, Barry et al. recorded the firing of place cells in the brain of the animals when they were on the track and during a rest period afterwards. The rats were then allowed onto the inaccessible arms, and again their brain activity was recorded.

In the rest period after the rats first viewed the inaccessible arms, the place cell pattern that would later form the mental map of a journey to and from the food-containing arm was pre-activated. However, the place cell pattern that would become the mental map of the other inaccessible arm was not activated before the rat explored that area. Therefore, Ólafsdóttir, Barry et al. suggest that the perception of reward influences which place cell pattern is simulated during rest. An implication of these findings is that the brain preferentially simulates past or future experiences that are deemed to be functionally significant, such as those associated with reward. A future challenge will be to determine whether this goal-related simulation of unvisited spaces predicts and is needed for behaviour such as successful navigation to a goal.

DOI:http://dx.doi.org/10.7554/eLife.06063.002

Self-organization of synchronous activity propagation in neuronal networks driven by local excitation

Many experimental and theoretical studies have suggested that the reliable propagation of synchronous neural activity is crucial for neural information processing. The propagation of synchronous firing activity in so-called synfire chains has been studied extensively in feed-forward networks of spiking neurons. However, it remains unclear how such neural activity could emerge in recurrent neuronal networks through synaptic plasticity. In this study, we investigate whether local excitation, i.e., neurons that fire at a higher frequency than the other, spontaneously active neurons in the network, can shape a network to allow for synchronous activity propagation. We use two-dimensional, locally connected and heterogeneous neuronal networks with spike-timing dependent plasticity (STDP). We find that, in our model, local excitation drives profound network changes within seconds. In the emergent network, neural activity propagates synchronously through the network. This activity originates from the site of the local excitation and propagates through the network. The synchronous activity propagation persists, even when the local excitation is removed, since it derives from the synaptic weight matrix. Importantly, once this connectivity is established it remains stable even in the presence of spontaneous activity. Our results suggest that synfire-chain-like activity can emerge in a relatively simple way in realistic neural networks by locally exciting the desired origin of the neuronal sequence.

Not only … but also: REM sleep creates and NREM Stage 2 instantiates landmark junctions in cortical memory networks

This article argues both rapid eye movement (REM) and non-rapid eye movement (NREM) sleep contribute to overnight episodic memory processes but their roles differ. Episodic memory may have evolved from memory for spatial navigation in animals and humans. Equally, mnemonic navigation in world and mental space may rely on fundamentally equivalent processes. Consequently, the basic spatial network characteristics of pathways which meet at omnidirectional nodes or junctions may be conserved in episodic brain networks. A pathway is formally identified with the unidirectional, sequential phases of an episodic memory. In contrast, the function of omnidirectional junctions is not well understood. In evolutionary terms, both animals and early humans undertook tours to a series of landmark junctions, to take advantage of resources (food, water and shelter), whilst trying to avoid predators. Such tours required memory for emotionally significant landmark resource-place-danger associations and the spatial relationships amongst these landmarks. In consequence, these tours may have driven the evolution of both spatial and episodic memory. The environment is dynamic. Resource-place associations are liable to shift and new resource-rich landmarks may be discovered, these changes may require re-wiring in neural networks. To realise these changes, REM may perform an associative, emotional encoding function between memory networks, engendering an omnidirectional landmark junction which is instantiated in the cortex during NREM Stage 2. In sum, REM may preplay associated elements of past episodes (rather than replay individual episodes), to engender an unconscious representation which can be used by the animal on approach to a landmark junction in wake.

Evidence for consolidation of neuronal assemblies after seizures in humans

The establishment of memories involves reactivation of waking neuronal activity patterns and strengthening of associated neural circuits during slow-wave sleep (SWS), a process known as "cellular consolidation" (Dudai and Morris, 2013). Reactivation of neural activity patterns during waking behaviors that occurs on a timescale of seconds to minutes is thought to constitute memory recall (O'Keefe and Nadel, 1978), whereas consolidation of memory traces may be revealed and served by correlated firing (reactivation) that appears during sleep under conditions suitable for synaptic modification (Buhry et al., 2011). Although reactivation has been observed in human neuronal recordings (Gelbard-Sagiv et al., 2008; Miller et al., 2013), reactivation during sleep has not, likely because data are difficult to obtain and the effect is subtle. Seizures, however, provide intense and synchronous, yet sparse activation (Bower et al., 2012) that could produce a stronger consolidation effect if seizures activate learning-related mechanisms similar to those activated by learned tasks. Continuous wide-bandwidth recordings from patients undergoing intracranial monitoring for drug-resistant epilepsy revealed reactivation of seizure-related neuronal activity during subsequent SWS, but not wakefulness. Those neuronal assemblies that were most strongly activated during seizures showed the largest correlation changes, suggesting that consolidation selectively strengthened neuronal circuits activated by seizures. These results suggest that seizures "hijack" physiological learning mechanisms and also suggest a novel epilepsy therapy targeting neuronal dynamics during post-seizure sleep.

Decoding the cognitive map: ensemble hippocampal sequences and decision making

Tolman proposed that complex animal behavior is mediated by the cognitive map, an integrative learning system that allows animals to reconfigure previous experience in order to compute predictions about the future. The discovery of place cells in the rodent hippocampus immediately suggested a plausible neural mechanism to fulfill the 'map' component of Tolman's theory. Recent work examining hippocampal representations occurring at fast time scales suggests that these sequences might be important for supporting the inferential mental operations associated with the cognitive map function. New findings that hippocampal sequences play an important causal role in mediating adaptive behavior on a moment-by-moment basis suggest specific neural processes that may underlie Tolman's cognitive map framework.