Hippocampal Mismatch Signals Are Modulated by the Strength of Neural Predictions and Their Similarity to Outcomes

Same Same, But Different: Brain Areas Underlying the Learning from Repetitive Episodic Prediction Errors

Prediction errors (PEs) function as learning signals. It is yet unclear how varying compared to repetitive PEs affect episodic memory in brain and behavior. The current study investigated cerebral and behavioral effects of experiencing either multiple alternative versions ("varying") or one single alternative version ("repetitive") of a previously encoded episode. Participants encoded a set of episodes ("originals") by watching videos showing toy stories. During scanning, participants either experienced originals, one single, or multiple alternative versions of the previously encoded episodes. Participants' memory performance was tested through recall of original objects. Varying and repetitive PEs revealed typical brain responses to the detection of mismatching information including inferior frontal and posterior parietal regions, as well as hippocampus, which is further linked to memory reactivation, and the amygdala, known for modulating memory consolidation. Furthermore, experiencing varying and repetitive PEs triggered distinct brain areas as revealed by direct contrast. Among others, experiencing varying versions triggered activity in the caudate, a region that has been associated with PEs. In contrast, repetitive PEs activated brain areas that resembled more those for retrieval of originally encoded episodes. Thus, ACC and posterior cingulate cortex activation seemed to serve both reactivating old and integrating new but similar information in episodic memory. Consistent with neural findings, participants recalled original objects less accurately when only presented with the same, but not varying, PE during fMRI. The current findings suggest that repeated PEs interact more strongly with a recalled original episodic memory than varying PEs.

A response time model of the three-choice Mnemonic Similarity Task provides stable, mechanistically interpretable individual-difference measures

Introduction

The Mnemonic Similarity Task (MST) is a widely used measure of individual tendency to discern small differences between remembered and presently presented stimuli. Significant work has established this measure as a reliable index of neurological and cognitive dysfunction and decline. However, questions remain about the neural and psychological mechanisms that support performance in the task.

Methods

Here, we provide new insights into these questions by fitting seven previously-collected MST datasets (total N = 519), adapting a three-choice evidence accumulation model (the Linear Ballistic Accumulator). The model decomposes choices into automatic and deliberative components.

Results

We show that these decomposed processes both contribute to the standard measure of behavior in this task, as well as capturing individual variation in this measure across the lifespan. We also exploit a delayed test/re-test manipulation in one of the experiments to show that model parameters exhibit improved stability, relative to the standard metric, across a 1 week delay. Finally, we apply the model to a resting-state fMRI dataset, finding that only the deliberative component corresponds to off-task co-activation in networks associated with long-term, episodic memory.

Discussion

Taken together, these findings establish a novel mechanistic decomposition of MST behavior and help to constrain theories about the cognitive processes that support performance in the task.

Neural correlates of temporal recalibration to delayed auditory feedback of active and passive movements

When we perform an action, its sensory outcomes usually follow shortly after. This characteristic temporal relationship aids in distinguishing self- from externally generated sensory input. To preserve this ability under dynamically changing environmental conditions, our expectation of the timing between action and outcome must be able to recalibrate, for example, when the outcome is consistently delayed. Until now, it remains unclear whether this process, known as sensorimotor temporal recalibration, can be specifically attributed to recalibration of sensorimotor (action-outcome) predictions, or whether it may be partly due to the recalibration of expectations about the intersensory (e.g., audio-tactile) timing. Therefore, we investigated the behavioral and neural correlates of temporal recalibration and differences in sensorimotor and intersensory contexts. During fMRI, subjects were exposed to delayed or undelayed tones elicited by actively or passively generated button presses. While recalibration of the expected intersensory timing (i.e., between the tactile sensation during the button movement and the tones) can be expected to occur during both active and passive movements, recalibration of sensorimotor predictions should be limited to active movement conditions. Effects of this procedure on auditory temporal perception and the modality-transfer to visual perception were tested in a delay detection task. Across both contexts, we found recalibration to be associated with activations in hippocampus and cerebellum. Context-dependent differences emerged in terms of stronger behavioral recalibration effects in sensorimotor conditions and were captured by differential activation pattern in frontal cortices, cerebellum, and sensory processing regions. These findings highlight the role of the hippocampus in encoding and retrieving newly acquired temporal stimulus associations during temporal recalibration. Furthermore, recalibration-related activations in the cerebellum may reflect the retention of multiple representations of temporal stimulus associations across both contexts. Finally, we showed that sensorimotor predictions modulate recalibration-related processes in frontal, cerebellar, and sensory regions, which potentially account for the perceptual advantage of sensorimotor versus intersensory temporal recalibration.

Long-term, multi-event surprise correlates with enhanced autobiographical memory

Neurobiological and psychological models of learning emphasize the importance of prediction errors (surprises) for memory formation. This relationship has been shown for individual momentary surprising events; however, it is less clear whether surprise that unfolds across multiple events and timescales is also linked with better memory of those events. We asked basketball fans about their most positive and negative autobiographical memories of individual plays, games and seasons, allowing surprise measurements spanning seconds, hours and months. We used advanced analytics on National Basketball Association play-by-play data and betting odds spanning 17 seasons, more than 22,000 games and more than 5.6 million plays to compute and align the estimated surprise value of each memory. We found that surprising events were associated with better recall of positive memories on the scale of seconds and months and negative memories across all three timescales. Game and season memories could not be explained by surprise at shorter timescales, suggesting that long-term, multi-event surprise correlates with memory. These results expand notions of surprise in models of learning and reinforce its relevance in real-world domains.

Minor Changes Change Memories: Functional Magnetic Resonance Imaging and Behavioral Reflections of Episodic Prediction Errors

Episodic memories can be modified, a process that is potentially driven by mnemonic prediction errors. In the present study, we used modified cues to induce prediction errors of different episodic relevance. Participants encoded episodes in the form of short toy stories and then returned for an fMRI session on the subsequent day. Here, participants were presented either original episodes or slightly modified versions thereof. Modifications consisted of replacing a single object within the episode and either challenged the gist of an episode (gist modifications) or left it intact (surface modifications). On the next day, participants completed a post-fMRI memory test that probed memories for originally encoded episodes. Both types of modifications triggered brain activation in regions we previously found to be involved in the processing of content-based mnemonic prediction errors (i.e., the exchange of an object). Specifically, these were ventrolateral pFC, intraparietal cortex, and lateral occipitotemporal cortex. In addition, gist modifications triggered pronounced brain responses, whereas those for surface modification were only significant in the right inferior frontal sulcus. Processing of gist modifications also involved the posterior temporal cortex and the precuneus. Interestingly, our findings confirmed the posterior hippocampal role of detail processing in episodic memory, as evidenced by increased posterior hippocampal activity for surface modifications compared with gist modifications. In the post-fMRI memory test, previous experience with surface modified, but not gist-modified episodes, increased erroneous acceptance of the same modified versions as originally encoded. Whereas surface-level prediction errors might increase uncertainty and facilitate confusion of alternative episode representations, gist-level prediction errors seem to trigger the clear distinction of independent episodes.

Spatiotemporal Dynamics of Memory Encoding and Memory Retrieval States

Memory encoding and memory retrieval are neurally distinct brain states that can be differentiated on the basis of cortical network activity. However, it is unclear whether sustained engagement of one network or fluctuations between multiple networks give rise to these memory states. The spatiotemporal dynamics of memory states may have important implications for memory behavior and cognition; however, measuring temporally resolved signals of cortical networks poses a challenge. Here, we recorded scalp electroencephalography from participants performing a mnemonic state task in which they were biased toward memory encoding or retrieval. We performed a microstate analysis to measure the temporal dynamics of cortical networks throughout this mnemonic state task. We find that Microstate E, a putative analog of the default mode network, shows temporally sustained dissociations between memory encoding and retrieval, with greater engagement during retrieve compared with encode trials. We further show that decreased engagement of Microstate E is a general property of encoding, rather than a reflection of retrieval suppression. Thus, memory success, as well as cognition more broadly, may be influenced by the ability to engage or disengage Microstate E in a goal-dependent manner.

The effect of prediction error on episodic memory encoding is modulated by the outcome of the predictions

Expectations can lead to prediction errors of varying degrees depending on the extent to which the information encountered in the environment conforms with prior knowledge. While there is strong evidence on the computationally specific effects of such prediction errors on learning, relatively less evidence is available regarding their effects on episodic memory. Here, we had participants work on a task in which they learned context/object-category associations of different strengths based on the outcomes of their predictions. We then used a reinforcement learning model to derive subject-specific trial-to-trial estimates of prediction error at encoding and link it to subsequent recognition memory. Results showed that model-derived prediction errors at encoding influenced subsequent memory as a function of the outcome of participants' predictions (correct vs. incorrect). When participants correctly predicted the object category, stronger prediction errors (as a consequence of weak expectations) led to enhanced memory. In contrast, when participants incorrectly predicted the object category, stronger prediction errors (as a consequence of strong expectations) led to impaired memory. These results highlight the important moderating role of choice outcome that may be related to interactions between the hippocampal and striatal dopaminergic systems.

Solidity Meets Surprise: Cerebral and Behavioral Effects of Learning from Episodic Prediction Errors

How susceptible a memory is to later modification might depend on how stable the episode has been encoded. This stability was proposed to increase when retrieving information more (vs. less) often and in a spaced (vs. massed) practice. Using fMRI, we examined the effects of these different pre-fMRI retrieval protocols on the subsequent propensity to learn from episodic prediction errors. After encoding a set of different action stories, participants came back for two pre-fMRI retrieval sessions in which they encountered original episodes either 2 or 8 times in either a spaced or a massed retrieval protocol. One week later, we cued episodic retrieval during the fMRI session by using original or modified videos of encoded action stories. Recurrent experience of modified episodes was associated with increasing activity in the episodic memory network including hippocampal and cortical areas, when leading to false memories in a post-fMRI memory test. While this observation clearly demonstrated learning from episodic prediction errors, we found no evidence for a modulatory effect of the different retrieval protocols. As expected, the benefit of retrieving an episode more often was reflected in better memory for originally encoded episodes. In addition, frontal activity increased for episodic prediction errors when episodes had been less frequently retrieved pre-fMRI. A history of spaced versus massed retrieval was associated with increased activation throughout the episodic memory network, with no significant effect on behavioral performance. Our findings show that episodic prediction errors led to false memories. The history of different retrieval protocols was reflected in memory performance and brain responses to episodic prediction errors, but did not interact with the brain's episodic learning response.

Experiencing Surprise: The Temporal Dynamics of Its Impact on Memory

To efficiently process information, the brain shifts between encoding and retrieval states, prioritizing bottom-up or top-down processing accordingly. Expectation violation before or during learning has been shown to trigger an adaptive encoding mechanism, resulting in better memory for unexpected events. Using fMRI, we explored (1) whether this encoding mechanism is also triggered during retrieval, and if so, (2) what the temporal dynamics of its mnemonic consequences are. Male and female participants studied object images, then, with new objects, they learned a contingency between a cue and a semantic category. Rule-abiding (expected) and violating (unexpected) targets and similar foils were used at test. We found interactions between previous and current similar events' expectation, such that when an expected event followed a similar but unexpected event, its performance was boosted, underpinned by activation in the hippocampus, midbrain, and occipital cortex. In contrast, a sequence of two unexpected similar events also triggered occipital engagement; however, this did not enhance memory performance. Taken together, our findings suggest that when the goal is to retrieve, encountering surprising events engages an encoding mechanism, supported by bottom-up processing, that may enhance memory for future related events.SIGNIFICANCE STATEMENT Optimizing the balance between new learning and the retrieval of existing knowledge is an ongoing process, at the core of human cognition. Previous research into memory encoding suggests experiencing surprise leads to the prioritization of the learning of new memories, forming an adaptive encoding mechanism. We examined whether this mechanism is also engaged when the current goal is to retrieve information. Our results demonstrate that an expectation-driven shift toward an encoding state, supported by enhanced perceptual processing, is beneficial for the correct identification of subsequent expected similar events. These findings have important implications for our understanding of the temporal dynamics of the adaptive encoding of information into memory.

Hippocampal representations switch from errors to predictions during acquisition of predictive associations

We constantly exploit the statistical regularities in our environment to help guide our perception. The hippocampus has been suggested to play a pivotal role in both learning environmental statistics, as well as exploiting them to generate perceptual predictions. However, it is unclear how the hippocampus balances encoding new predictive associations with the retrieval of existing ones. Here, we present the results of two high resolution human fMRI studies (N = 24 for both experiments) directly investigating this. Participants were exposed to auditory cues that predicted the identity of an upcoming visual shape (with 75% validity). Using multivoxel decoding analysis, we find that the hippocampus initially preferentially represents unexpected shapes (i.e., those that violate the cue regularities), but later switches to representing the cue-predicted shape regardless of which was actually presented. These findings demonstrate that the hippocampus is involved both acquiring and exploiting predictive associations, and is dominated by either errors or predictions depending on whether learning is ongoing or complete.

Lower functional hippocampal connectivity in healthy adults is jointly associated with higher levels of leptin and insulin resistance

BACKGROUND

Metabolic dysregulation is currently considered a major risk factor for hippocampal pathology. The aim of the present study was to characterize the influence of key metabolic drivers on functional connectivity of the hippocampus in healthy adults.

METHODS

Insulin resistance was directly quantified by measuring steady-state plasma glucose (SSPG) concentration during the insulin suppression test and fasting levels of insulin, glucose, leptin, and cortisol, and measurements of body mass index and waist circumference were obtained in a sample of healthy cognitively intact adults (n = 104). Resting-state neuroimaging data were also acquired for the quantification of hippocampal functional cohesiveness and integration with the major resting-state networks (RSNs). Data-driven analysis using unsupervised machine learning (k-means clustering) was then employed to identify clusters of individuals based on their metabolic and functional connectivity profiles.

RESULTS

K-means clustering identified two clusters of increasing metabolic deviance evidenced by cluster differences in the plasma levels of leptin (40.36 (29.97) vs. 27.59 (25.58) μg/L) and the degree of insulin resistance (SSPG concentration: 161.63 (65.27) vs. 125.72 (66.81) mg/dL). Individuals in the cluster with higher metabolic deviance showed lower functional cohesiveness within each hippocampus and lower integration of posterior and anterior components of the left and right hippocampus with the major RSNs. The two clusters did not differ in general intellectual ability or episodic memory.

CONCLUSIONS

We identified two clusters of individuals differentiated by abnormalities in insulin resistance, leptin levels, and hippocampal connectivity, with one of the clusters showing greater deviance. These findings support the link between metabolic dysregulation and hippocampal function even in nonclinical samples.

What Happened When? Cerebral Processing of Modified Structure and Content in Episodic Cueing

Episodic memories are not static but can change on the basis of new experiences, potentially allowing us to make valid predictions in the face of an ever-changing environment. Recent research has identified prediction errors during memory retrieval as a possible trigger for such changes. In this study, we used modified episodic cues to investigate whether different types of mnemonic prediction errors modulate brain activity and subsequent memory performance. Participants encoded episodes that consisted of short toy stories. During a subsequent fMRI session, participants were presented videos showing the original episodes, or slightly modified versions thereof. In modified videos, either the order of two subsequent action steps was changed or an object was exchanged for another. Content modifications recruited parietal, temporo-occipital, and parahippocampal areas reflecting the processing of the new object information. In contrast, structure modifications elicited activation in right dorsal premotor, posterior temporal, and parietal areas, reflecting the processing of new sequence information. In a post-fMRI memory test, the participants' tendency to accept modified episodes as originally encoded increased significantly when they had been presented modified versions already during the fMRI session. After experiencing modifications, especially those of the episodes' structure, the recognition of originally encoded episodes was impaired as well. Our study sheds light onto the neural processing of different types of episodic prediction errors and their influence on subsequent memory recall.

A neural network model of when to retrieve and encode episodic memories

Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.

Seeing What I Did (Not): Cerebral and Behavioral Effects of Agency and Perspective on Episodic Memory Re-activation

Intuitively, we assume that we remember episodes better when we actively participated in them and were not mere observers. Independently of this, we can recall episodes from either the first-person perspective (1pp) or the third-person perspective (3pp). In this functional magnetic resonance imaging (fMRI) study, we tested whether agency and perspective modulate neural activity during memory retrieval and subsequently enhance memory performance. Subjects encoded a set of different episodes by either imitating or only observing videos that showed short toy stories. A week later, we conducted fMRI and cued episodic retrieval by presenting the original videos, or slightly modified versions thereof, from 1pp or from 3pp. The hippocampal formation was sensitive to self-performed vs. only observed actions only when there was an episodic mismatch. In a post-fMRI memory test a history of self-performance did not improve behavioral memory performance. However, modified videos were often (falsely) accepted as showing truly experienced episodes when: (i) they were already presented in this modified version during fMRI or (ii) they were presented in their original form during fMRI but from 3pp. While the overall effect of modification was strong, the effects of perspective and agency were more subtle. Together, our findings demonstrate that self-performance and self-perspective modulate the strength of a memory trace in different ways. Even when memory performance remains the same for different agentive states, the brain is capable of detecting mismatching information. Re-experiencing the latter impairs memory performance as well as retrieving encoded episodes from 3pp.

Hippocampal volume reduction is associated with direct measure of insulin resistance in adults

Hippocampal integrity is highly susceptible to metabolic dysfunction, yet its mechanisms are not well defined. We studied 126 healthy individuals aged 23-61 years. Insulin resistance (IR) was quantified by measuring steady-state plasma glucose (SSPG) concentration during the insulin suppression test. Body mass index (BMI), adiposity, fasting insulin, glucose, leptin as well as structural neuroimaing with automatic hippocampal subfield segmentation were performed. Data analysis using unsupervised machine learning (k-means clustering) identified two subgroups reflecting a pattern of more pronounced hippocampal volume reduction being concurrently associated with greater adiposity and insulin resistance; the hippocampal volume reductions were uniform across subfields. Individuals in the most deviant subgroup were predominantly women (79 versus 42 %) with higher BMI [27.9 (2.5) versus 30.5 (4.6) kg/m²], IR (SSPG concentration, [156 (61) versus 123 (70) mg/dL] and leptinemia [21.7 (17.0) versus 44.5 (30.4) μg/L]. The use of person-based modeling in healthy individuals suggests that adiposity, insulin resistance and compromised structural hippocampal integrity behave as a composite phenotype; female sex emerged as risk factor for this phenotype.

Mnemonic prediction errors promote detailed memories

When our experience violates our predictions, it is adaptive to update our knowledge to promote a more accurate representation of the world and facilitate future predictions. Theoretical models propose that these mnemonic prediction errors should be encoded into a distinct memory trace to prevent interference with previous, conflicting memories. We investigated this proposal by repeatedly exposing participants to pairs of sequentially presented objects (A → B), thus evoking expectations. Then, we violated participants' expectations by replacing the second object in the pairs with a novel object (A → C). The following item memory test required participants to discriminate between identical old items and similar lures, thus testing detailed and distinctive item memory representations. In two experiments, mnemonic prediction errors enhanced item memory: Participants correctly identified more old items as old when those items violated expectations during learning, compared with items that did not violate expectations. This memory enhancement for C items was only observed when participants later showed intact memory for the related A → B pairs, suggesting that strong predictions are required to facilitate memory for violations. Following up on this, a third experiment reduced prediction strength prior to violation and subsequently eliminated the memory advantage of violations. Interestingly, mnemonic prediction errors did not increase gist-based mistakes of identifying old items as similar lures or identifying similar lures as old. Enhanced item memory in the absence of gist-based mistakes suggests that violations enhanced memory for items' details, which could be mediated via distinct memory traces. Together, these results advance our knowledge of how mnemonic prediction errors promote memory formation.

The Prefrontal-Hippocampal Comparator: Volition and Episodic Memory

This review describes recent research that is relevant to the prefrontal-hippocampal comparator model with the following conclusions: 1. Hippocampal area CA1 serves, at least in part, as an associative match-mismatch comparator. 2. Voluntary movement strengthens episodic memories for goal-directed behavior. 3. Hippocampal theta power serves as a prediction error signal during hippocampal dependent tasks. 4. The self-referential component of episodic memory in humans is mediated by the corollary discharge (the efference copy of the action plan developed by prefrontal cortex and transmitted to hippocampus where it is stored as a working memory; CA1 uses this efference copy to compare the expected consequences of action to the actual consequences of action). 5. Impairments in the production or transmission of this corollary discharge may contribute to some of the symptoms of schizophrenia. Unresolved issues and suggestions for future research are discussed.

Expectation-driven novelty effects in episodic memory

Novel and unexpected stimuli are often prioritised in memory, given their inherent salience. Nevertheless, not all forms of novelty show such an enhancement effect. Here, we discuss the role expectation plays in modulating the way novelty affects memory processes, circuits, and subsequent performance. We first review independent effects of expectation on memory, and then consider how different types of novelty are characterised by expectation. We argue that different types of novelty defined by expectation implicate differential neurotransmission in memory formation brain regions and may also result in the creation of different types of memory. Contextual novelty, which is unexpected by definition, is often associated with better recollection, supported by dopaminergic-hippocampal interactions. On the other hand, expected stimulus novelty is supported by engagement of medial temporal cortices, as well as the hippocampus, through cholinergic modulation. Furthermore, when expected stimulus novelty results in enhanced memory, it is predominantly driven by familiarity. The literature reviewed here highlights the complexity of novelty-sensitive memory systems, the distinction between types of novelty, and how they are differentially affected by expectancy.

Adaptive Memory Distortions Are Predicted by Feature Representations in Parietal Cortex

Similarity between memories is a primary cause of interference and forgetting. Exaggerating subtle differences between memories is therefore a potential mechanism for reducing interference. Here, we report a human fMRI study (n = 29, 19 female) that tested whether behavioral and neural expressions of memories are adaptively distorted to reduce interference. Participants learned and repeatedly retrieved object images, some of which were identical except for subtle color differences. Behavioral measures of color memory revealed exaggeration of differences between similar objects. Importantly, greater memory exaggeration was associated with lower memory interference. fMRI pattern analyses revealed that color information in parietal cortex was stronger during memory recall when color information was critical for discriminating competing memories. Moreover, greater representational distance between competing memories in parietal cortex predicted greater color memory exaggeration and lower memory interference. Together, these findings reveal that competition between memories induces adaptive, feature-specific distortions in parietal representations and corresponding behavioral expressions.SIGNIFICANCE STATEMENT Similarity between memories is a primary cause of interference and forgetting. Here, we show that, when remembering highly similar objects, subtle differences in the features of these objects are exaggerated in memory to reduce interference. These memory distortions are reflected in, and predicted by, overlap of activity patterns in lateral parietal cortex. These findings provide unique insight into how memory interference is resolved and specifically implicate lateral parietal cortex in representing feature-specific memory distortions.

Cortical Representations of Visual Stimuli Shift Locations with Changes in Memory States

Episodic memory retrieval is thought to rely on reactivation of the same content-sensitive neural activity patterns initially expressed during memory encoding.^1-6 Yet there are emerging examples of content representations expressed in different brain regions during encoding versus retrieval.^7-14 Although these differences have been observed by comparing encoding and retrieval tasks that differ in terms of perceptual experience and cognitive demands, there are many real-world contexts-e.g., meeting a new colleague who reminds you of an old acquaintance-where the memory system might be intrinsically biased either toward encoding (the new colleague) or retrieval (the old acquaintance).¹⁵¹⁶ Here, we test whether intrinsic memory states, independent of task demands, determine the cortical location of content representations. In a human fMRI study, subjects (n = 33) viewed object images and were instructed to either encode the current object or retrieve a similar object from memory. Using pattern classifiers, we show that biases toward encoding versus retrieval were reflected in large-scale attentional networks.^17-19 Critically, memory states decoded from these networks-even when entirely independent from task instructions-predicted shifts of object representations from visual cortex (encoding) to ventral parietal cortex (retrieval). Finally, visual versus ventral parietal cortices exhibited differential connectivity with the hippocampus during memory encoding versus retrieval, consistent with the idea that the hippocampus mediates cortical shifts in content representations. Collectively, these findings demonstrate that intrinsic biases toward memory encoding versus retrieval determine the specific cortical locations that express content information.

Transforming the Concept of Memory Reactivation

Reactivation refers to the phenomenon wherein patterns of neural activity expressed during perceptual experience are re-expressed at a later time, a putative neural marker of memory. Reactivation of perceptual content has been observed across many cortical areas and correlates with objective and subjective expressions of memory in humans. However, because reactivation emphasizes similarities between perceptual and memory-based representations, it obscures differences in how perceptual events and memories are represented. Here, we highlight recent evidence of systematic differences in how (and where) perceptual events and memories are represented in the brain. We argue that neural representations of memories are best thought of as spatially transformed versions of perceptual representations. We consider why spatial transformations occur and identify critical questions for future research.

Hippocampus plays a role in speech feedback processing

There is increasing evidence that the hippocampus is involved in language production and verbal communication, although little is known about its possible role. According to one view, hippocampus contributes semantic memory to spoken language. Alternatively, hippocampus is involved in the processing the (mis)match between expected sensory consequences of speaking and the perceived speech feedback. In the current study, we re-analysed functional magnetic resonance (fMRI) data of two overt picture-naming studies to test whether hippocampus is involved in speech production and, if so, whether the results can distinguish between a "pure memory" versus a "prediction" account of hippocampal involvement. In both studies, participants overtly named pictures during scanning while hearing their own speech feedback unimpededly or impaired by a superimposed noise mask. Results showed decreased hippocampal activity when speech feedback was impaired, compared to when feedback was unimpeded. Further, we found increased functional coupling between auditory cortex and hippocampus during unimpeded speech feedback, compared to impaired feedback. Finally, we found significant functional coupling between a hippocampal/supplementary motor area (SMA) interaction term and auditory cortex, anterior cingulate cortex and cerebellum during overt picture naming, but not during listening to one's own pre-recorded voice. These findings indicate that hippocampus plays a role in speech production that is in accordance with a "prediction" view of hippocampal functioning.

Hippocampal theta coordinates memory processing during visual exploration

The hippocampus supports memory encoding and retrieval, which may occur at distinct phases of the theta cycle. These processes dynamically interact over rapid timescales, especially when sensory information conflicts with memory. The ability to link hippocampal dynamics to memory-guided behaviors has been limited by experiments that lack the temporal resolution to segregate encoding and retrieval. Here, we simultaneously tracked eye movements and hippocampal field potentials while neurosurgical patients performed a spatial memory task. Phase-locking at the peak of theta preceded fixations to retrieved locations, indicating that the hippocampus coordinates memory-guided eye movements. In contrast, phase-locking at the trough of theta followed fixations to novel object-locations and predicted intact memory of the original location. Theta-gamma phase amplitude coupling increased during fixations to conflicting visual content, but predicted memory updating. Hippocampal theta thus supports learning through two interleaved processes: strengthening encoding of novel information and guiding exploration based on prior experience.

Pattern Separation Underpins Expectation-Modulated Memory

Pattern separation and completion are fundamental hippocampal computations supporting memory encoding and retrieval. However, despite extensive exploration of these processes, it remains unclear whether and how top-down processes adaptively modulate the dynamics between these computations. Here we examine the role of expectation in shifting the hippocampus to perform pattern separation. In a behavioral task, 29 participants (7 males) learned a cue-object category contingency. Then, at encoding, one-third of the cues preceding the to-be-memorized objects, violated the studied rule. At test, participants performed a recognition task with old objects (targets) and a set of parametrically manipulated (very similar to dissimilar) foils for each object. Accuracy was found to be better for foils of high similarity to targets that were contextually unexpected at encoding compared with expected ones. Critically, there were no expectation-driven differences for targets and low similarity foils. To further explore these effects, we implemented a computational model of the hippocampus, performing the same task as the human participants. We used representational similarity analysis to examine how top-down expectation interacts with bottom-up perceptual input, in each layer. All subfields showed more dissimilar representations for unexpected items, with dentate gyrus (DG) and CA3 being more sensitive to expectation violation than CA1. Again, representational differences between expected and unexpected inputs were prominent for moderate to high levels of input similarity. This effect diminished when inputs from DG and CA3 into CA1 were lesioned. Overall, these novel findings strongly suggest that pattern separation in DG/CA3 underlies the effect that violation of expectation exerts on memory.SIGNIFICANCE STATEMENT What makes some events more memorable than others is a key question in cognitive neuroscience. Violation of expectation often leads to better memory performance, but the neural mechanism underlying this benefit remains elusive. In a behavioral study, we found that memory accuracy is enhanced selectively for unexpected highly similar foils, suggesting expectation violation does not enhance memory indiscriminately, but specifically aids the disambiguation of overlapping inputs. This is further supported by our subsequent investigation using a hippocampal computational model, revealing increased representational dissimilarity for unexpected highly similar foils in DG and CA3. These convergent results provide the first evidence that pattern separation plays an explicit role in supporting memory for unexpected information.

Transformation of Event Representations along Middle Temporal Gyrus

When learning about events through visual experience, one must not only identify which events are visually similar but also retrieve those events' associates-which may be visually dissimilar-and recognize when different events have similar predictive relations. How are these demands balanced? To address this question, we taught participants the predictive structures among four events, which appeared in four different sequences, each cued by a distinct object. In each, one event ("cause") was predictably followed by another ("effect"). Sequences in the same relational category had similar predictive structure, while across categories, the effect and cause events were reversed. Using functional magnetic resonance imaging data, we measured "associative coding," indicated by correlated responses between effect and cause events; "perceptual coding," indicated by correlated responses to visually similar events; and "relational category coding," indicated by correlated responses to sequences in the same relational category. All three models characterized responses within the right middle temporal gyrus (MTG), but in different ways: Perceptual and associative coding diverged along the posterior to anterior axis, while relational categories emerged anteriorly in tandem with associative coding. Thus, along the posterior-anterior axis of MTG, the representation of the visual attributes of events is transformed to a representation of both specific and generalizable relational attributes.

Predictability Changes What We Remember in Familiar Temporal Contexts

The human brain constantly anticipates the future based on memories of the past. Encountering a familiar situation reactivates memory of previous encounters, which can trigger a prediction of what comes next to facilitate responsiveness. However, a prediction error can lead to pruning of the offending memory, a process that weakens its representation in the brain and leads to forgetting. Our goal in this study was to evaluate whether memories are spared from such pruning in situations that allow for accurate predictions at the categorical level, despite prediction errors at the item level. Participants viewed a sequence of objects, some of which reappeared multiple times ("cues"), followed always by novel items. Half of the cues were followed by new items from different (unpredictable) categories, while others were followed by new items from a single (predictable) category. Pattern classification of fMRI data was used to identify category-specific predictions after each cue. Pruning was observed only in unpredictable contexts, while encoding of new items was less robust in predictable contexts. These findings demonstrate that how associative memories are updated is influenced by the reliability of abstract-level predictions in familiar contexts.

Interference between overlapping memories is predicted by neural states during learning

One of the primary contributors to forgetting is interference from overlapping memories. Intuitively, this suggests-and prominent theoretical models argue-that memory interference is best avoided by encoding overlapping memories as if they were unrelated. It is therefore surprising that reactivation of older memories during new encoding has been associated with reduced memory interference. Critically, however, prior studies have not directly established why reactivation reduces interference. Here, we first developed a behavioral paradigm that isolates the negative influence that overlapping memories exert during memory retrieval. We then show that reactivating older memories during the encoding of new memories dramatically reduces this interference cost at retrieval. Finally, leveraging multiple fMRI decoding approaches, we show that spontaneous reactivation of older memories during new encoding leads to integration of overlapping memories and, critically, that integration during encoding specifically reduces interference between overlapping, and otherwise competing, memories during retrieval.

Role of prediction error in destabilizing fear memories in retrieval extinction and its neural mechanisms

Memory reconsolidation interference has been shown to be an effective way to neutralize conditioned fear memory and prevent relapse. The critical factor to utilize this paradigm is inducing a labile state of the long-term memory. Novel information is viewed as a driving factor to update memory; however, it is unknown whether different forms of novelty play the same role. In addition, although pharmacological intervention studies have confirmed that prediction error (PE) during reactivation is a necessary condition in memory destabilization, the role of PE in retrieval extinction has remained under debate; furthermore, the neural mechanisms underlying the process are largely unknown. In this study, we isolated two forms of novelty: PE and stimulus novelty without PE during reactivation to compare their role in memory lability. Skin conductance responses (SCR) and functional magnetic resonance imaging (fMRI) were used to clarify their role at the behavioural and neural mechanism levels. A total of 54 healthy adults were tested in a three-day retrieval extinction protocol. The results showed that PE, the novelty of CS-US combinations, was a critical condition to destabilize memory. The novelty of the stimulus itself with the absence of PE was insufficient for retrieving the memory. The neural mechanisms that distinguished standard extinction from retrieval extinction were that the latter was associated with a diminished recruitment of the inferior temporal cortex (IT) and dorsolateral prefrontal cortex (dlPFC) and decreased functional connectivity of the dlPFC-anterior cingulate cortex (ACC) and IT-dlPFC. Possible interpretations were discussed.

Content Tuning in the Medial Temporal Lobe Cortex: Voxels that Perceive, Retrieve

How do we recall vivid details from our past based only on sparse cues? Research suggests that the phenomenological reinstatement of past experiences is accompanied by neural reinstatement of the original percept. This process critically depends on the medial temporal lobe (MTL). Within the MTL, perirhinal cortex (PRC) and parahippocampal cortex (PHC) are thought to support encoding and recall of objects and scenes, respectively, with the hippocampus (HC) serving as a content-independent hub. If the fidelity of recall indeed arises from neural reinstatement of perceptual activity, then successful recall should preferentially draw upon those neural populations within content-sensitive MTL cortex that are tuned to the same content during perception. We tested this hypothesis by having eighteen human participants undergo functional MRI (fMRI) while they encoded and recalled objects and scenes paired with words. Critically, recall was cued with the words only. While HC distinguished successful from unsuccessful recall of both objects and scenes, PRC and PHC were preferentially engaged during successful versus unsuccessful object and scene recall, respectively. Importantly, within PRC and PHC, this content-sensitive recall was predicted by content tuning during perception: Across PRC voxels, we observed a positive relationship between object tuning during perception and successful object recall, while across PHC voxels, we observed a positive relationship between scene tuning during perception and successful scene recall. Our results thus highlight content-based roles of MTL cortical regions for episodic memory and reveal a direct mapping between content-specific tuning during perception and successful recall.

Positive reward prediction errors during decision-making strengthen memory encoding

Dopamine is thought to provide reward prediction error signals to temporal lobe memory systems, but the role of these signals in episodic memory has not been fully characterized. Here we developed an incidental memory paradigm to (i) estimate the influence of reward prediction errors on the formation of episodic memories, (ii) dissociate this influence from surprise and uncertainty, (iii) characterize the role of temporal correspondence between prediction error and memoranda presentation and (iv) determine the extent to which this influence is dependent on memory consolidation. We found that people encoded incidental memoranda more strongly when they gambled for potential rewards. Moreover, the degree to which gambling strengthened encoding scaled with the reward prediction error experienced when memoranda were presented (and not before or after). This encoding enhancement was detectable within minutes and did not differ substantially after 24 h, indicating that it is not dependent on memory consolidation. These results suggest a computationally and temporally specific role for reward prediction error signalling in memory formation.

Medial Orbitofrontal Cortex, Dorsolateral Prefrontal Cortex, and Hippocampus Differentially Represent the Event Saliency

Two primary functions attributed to the hippocampus and prefrontal cortex (PFC) network are retaining the temporal and spatial associations of events and detecting deviant events. It is unclear, however, how these two functions converge into one mechanism. Here, we tested whether increased activity with perceiving salient events is a deviant detection signal or contains information about the event associations by reflecting the magnitude of deviance (i.e., event saliency). We also tested how the deviant detection signal is affected by the degree of anticipation. We studied regional neural activity when people watched a movie that had varying saliency of a novel or an anticipated flow of salient events. Using intracranial electroencephalography from 10 patients, we observed that high-frequency activity (50-150 Hz) in the hippocampus, dorsolateral PFC, and medial OFC tracked event saliency. We also observed that medial OFC activity was stronger when the salient events were anticipated than when they were novel. These results suggest that dorsolateral PFC and medial OFC, as well as the hippocampus, signify the saliency magnitude of events, reflecting the hierarchical structure of event associations.

Hippocampal contributions to serial-order memory

Our memories form a record not only of our experiences, but also of their temporal structure. Although memory for the temporal structure of experience likely relies on multiple neural systems, numerous studies have implicated the hippocampus in the encoding and retrieval of temporal information. This review evaluates the literature on hippocampal contributions to human serial-order memory from the perspective of three cognitive theories: associative chaining theory, positional-coding theory and retrieved-context theory. Evaluating neural findings through the lens of cognitive theories enables us to draw more incisive conclusions about the relations between brain and behavior.

A translational perspective on neural circuits of fear extinction: Current promises and challenges

Fear extinction is the well-known process of fear reduction through repeated re-exposure to a feared stimulus without the aversive outcome. The last two decades have witnessed a surge of interest in extinction learning. First, extinction learning is observed across species, and especially research on rodents has made great strides in characterising the physical substrate underlying extinction learning. Second, extinction learning is considered of great clinical significance since it constitutes a crucial component of exposure treatment. While effective in reducing fear responding in the short term, extinction learning can lose its grip, resulting in a return of fear (i.e., laboratory model for relapse of anxiety symptoms in patients). Optimization of extinction learning is, therefore, the subject of intense investigation. It is thought that the success of extinction learning is, at least partly, determined by the mismatch between what is expected and what actually happens (prediction error). However, while much of our knowledge about the neural circuitry of extinction learning and factors that contribute to successful extinction learning comes from animal models, translating these findings to humans has been challenging for a number of reasons. Here, we present an overview of what is known about the animal circuitry underlying extinction of fear, and the role of prediction error. In addition, we conducted a systematic literature search to evaluate the degree to which state-of-the-art neuroimaging methods have contributed to translating these findings to humans. Results show substantial overlap between networks in animals and humans at a macroscale, but current imaging techniques preclude comparisons at a smaller scale, especially in sub-cortical areas that are functionally heterogeneous. Moreover, human neuroimaging shows the involvement of numerous areas that are not typically studied in animals. Results obtained in research aimed to map the extinction circuit are largely dependent on the methods employed, not only across species, but also across human neuroimaging studies. Directions for future research are discussed.

Surprise and destabilize: prediction error influences episodic memory reconsolidation

Through the process of "reconsolidation," reminders can temporarily destabilize memories and render them vulnerable to change. Recent rodent research has proposed that prediction error, or the element of surprise, is a key component of this process; yet, this hypothesis has never before been extended to complex episodic memories in humans. In our novel paradigm, we used naturalistic stimuli to demonstrate that prediction error enables adaptive updating of episodic memories. In Study 1, participants (N = 48) viewed 18 videos, each depicting an action-outcome event. The next day, we reactivated these memories by presenting the videos again. We found that incomplete reminders, which interrupted videos before the outcome, made memories vulnerable to subsequent interference from a new set of videos, producing false memories. In Study 2 (N = 408), an independent sample rated qualities of the stimuli. We found that videos that were more surprising when interrupted produced more false memories. Last, in Study 3 (N = 24), we tested competing predictions of reconsolidation theory and the Temporal Context Model, an alternative account of source confusion. Consistent with the mechanistic time-course of reconsolidation, our effects were crucially time-dependent. Overall, we synthesize prior animal and human research to present compelling evidence that prediction error destabilizes episodic memories and drives dynamic updating in the face of new information.

Parietal Representations of Stimulus Features Are Amplified during Memory Retrieval and Flexibly Aligned with Top-Down Goals

In studies of human episodic memory, the phenomenon of reactivation has traditionally been observed in regions of occipitotemporal cortex (OTC) involved in visual perception. However, reactivation also occurs in lateral parietal cortex (LPC), and recent evidence suggests that stimulus-specific reactivation may be stronger in LPC than in OTC. These observations raise important questions about the nature of memory representations in LPC and their relationship to representations in OTC. Here, we report two fMRI experiments that quantified stimulus feature information (color and object category) within LPC and OTC, separately during perception and memory retrieval, in male and female human subjects. Across both experiments, we observed a clear dissociation between OTC and LPC: while feature information in OTC was relatively stronger during perception than memory, feature information in LPC was relatively stronger during memory than perception. Thus, while OTC and LPC represented common stimulus features in our experiments, they preferentially represented this information during different stages. In LPC, this bias toward mnemonic information co-occurred with stimulus-level reinstatement during memory retrieval. In Experiment 2, we considered whether mnemonic feature information in LPC was flexibly and dynamically shaped by top-down retrieval goals. Indeed, we found that dorsal LPC preferentially represented retrieved feature information that addressed the current goal. In contrast, ventral LPC represented retrieved features independent of the current goal. Collectively, these findings provide insight into the nature and significance of mnemonic representations in LPC and constitute an important bridge between putative mnemonic and control functions of parietal cortex.SIGNIFICANCE STATEMENT When humans remember an event from the past, patterns of sensory activity that were present during the initial event are thought to be reactivated. Here, we investigated the role of lateral parietal cortex (LPC), a high-level region of association cortex, in representing prior visual experiences. We find that LPC contained stronger information about stimulus features during memory retrieval than during perception. We also found that current task goals influenced the strength of stimulus feature information in LPC during memory. These findings suggest that, in addition to early sensory areas, high-level areas of cortex, such as LPC, represent visual information during memory retrieval, and that these areas may play a special role in flexibly aligning memories with current goals.

Knowledge and learning of verb biases in amnesia

Verb bias-the co-occurrence frequencies between a verb and the syntactic structures it may appear with-is a critical and reliable linguistic cue for online sentence processing. In particular, listeners use this information to disambiguate sentences with multiple potential syntactic parses (e.g., Feel the frog with the feather.). Further, listeners dynamically update their representations of specific verbs in the face of new evidence about verb-structure co-occurrence. Yet, little is known about the biological memory systems that support the use and dynamic updating of verb bias. We propose that hippocampal-dependent declarative (relational) memory represents a likely candidate system because it has been implicated in the flexible binding of relational co-occurrences and in statistical learning. We explore this question by testing patients with severe and selective deficits in declarative memory (anterograde amnesia), and demographically matched healthy participants, in their on-line interpretation of ambiguous sentences and the ability to update their verb bias with experience. We find that (1) patients and their healthy counterparts use existing verb bias to successfully interpret on-line ambiguity, however (2) unlike healthy young adults, neither group updated these biases in response to recent exposure. These findings demonstrate that using existing representations of verb bias does not necessitate involvement of the declarative memory system, but leave open the question of whether the ability to update representations of verb-specific biases requires hippocampal engagement.

Bottom-Up and Top-Down Factors Differentially Influence Stimulus Representations Across Large-Scale Attentional Networks

Visual attention is thought to be supported by three large-scale frontoparietal networks: the frontoparietal control network (FPCN), the dorsal attention network (DAN), and the ventral attention network (VAN). The traditional view is that these networks support visual attention by biasing and evaluating sensory representations in visual cortical regions. However, recent evidence suggests that frontoparietal regions actively represent perceptual stimuli. Here, we assessed how perceptual stimuli are represented across large-scale frontoparietal and visual networks. Specifically, we tested whether representations of stimulus features across these networks are differentially sensitive to bottom-up and top-down factors. In a pair of pattern-based fMRI studies, male and female human subjects made perceptual decisions about face images that varied along two independent dimensions: gender and affect. Across studies, we interrupted bottom-up visual input using backward masks. Within studies, we manipulated which stimulus features were goal relevant (i.e., whether gender or affect was relevant) and task switching (i.e., whether the goal on the current trial matched the goal on the prior trial). We found that stimulus features could be reliably decoded from all four networks and, importantly, that subregions within each attentional network maintained coherent representations. Critically, the different attentional manipulations (interruption, goal relevance, and task switching) differentially influenced feature representations across networks. Whereas visual interruption had a relatively greater influence on representations in visual regions, goal relevance and task switching had a relatively greater influence on representations in frontoparietal networks. Therefore, large-scale brain networks can be dissociated according to how attention influences the feature representations that they maintain.SIGNIFICANCE STATEMENT Visual attention is supported by multiple frontoparietal attentional networks. However, it remains unclear how stimulus features are represented within these networks and how they are influenced by attention. Here, we assessed feature representations in four large-scale networks using a perceptual decision-making paradigm in which we manipulated top-down and bottom-up factors. We found that top-down manipulations such as goal relevance and task switching modulated feature representations in attentional networks, whereas bottom-up manipulations such as interruption of visual processing had a relatively stronger influence on feature representations in visual regions. Together, these findings indicate that attentional networks actively represent stimulus features and that representations within different large-scale networks are influenced by different forms of attention.

Consolidation Promotes the Emergence of Representational Overlap in the Hippocampus and Medial Prefrontal Cortex

Structured knowledge is thought to form, in part, through the extraction and representation of regularities across overlapping experiences. However, little is known about how consolidation processes may transform novel episodic memories to reflect such regularities. In a multi-day fMRI study, participants encoded trial-unique associations that shared features with other trials. Multi-variate pattern analyses were used to measure neural similarity across overlapping and non-overlapping memories during immediate and 1-week retrieval of these associations. We found that neural patterns in the hippocampus and medial prefrontal cortex represented the featural overlap across memories, but only after a week. Furthermore, after a week, the strength of a memory's unique episodic reinstatement during retrieval was inversely related to its representation of overlap, suggesting a trade-off between the integration of related memories and recovery of episodic details. These findings suggest that consolidation-related changes in neural representations support the gradual organization of discrete episodes into structured knowledge.

Memory integration constructs maps of space, time, and concepts

Recent evidence demonstrates that new events are learned in the context of their relationships to existing memories. Within the hippocampus and medial prefrontal cortex, related memories are represented by integrated codes that connect events experienced at different times and places. Integrated codes form the basis of spatial, temporal, and conceptual maps of experience. These maps represent information that goes beyond direct experience and support generalization behaviors that require knowledge be used in new ways. The degree to which an individual memory is integrated into a coherent map is determined by its spatial, temporal, and conceptual proximity to existing knowledge. Integration is observed over a wide range of scales, suggesting that memories contain information about both broad and fine-grained contexts.