Memory's penumbra: episodic memory decisions induce lingering mnemonic biases

Top-Down Task Goals Induce the Retrieval State

Engaging the retrieval state (Tulving, 1983) impacts processing and behavior (Long and Kuhl, 2019, 2021; Smith et al., 2022), but the extent to which top-down factors-explicit instructions and goals-versus bottom-up factors-stimulus properties such as repetition and similarity-jointly or independently induce the retrieval state is unclear. Identifying the impact of bottom-up and top-down factors on retrieval state engagement is critical for understanding how control of task-relevant versus task-irrelevant brain states influence cognition. We conducted between-subjects recognition memory tasks on male and female human participants in which we varied test phase goals. We recorded scalp electroencephalography and used an independently validated mnemonic state classifier (Long, 2023) to measure retrieval state engagement as a function of top-down task goals (recognize old vs detect new items) and bottom-up stimulus repetition (hits vs correct rejections (CRs)). We find that whereas the retrieval state is engaged for hits regardless of top-down goals, the retrieval state is only engaged during CRs when the top-down goal is to recognize old items. Furthermore, retrieval state engagement is greater for low compared to high confidence hits when the task goal is to recognize old items. Together, these results suggest that top-down demands to recognize old items induce the retrieval state independent from bottom-up factors, potentially reflecting the recruitment of internal attention to enable access of a stored representation.

Benefits of spaced learning are predicted by re-encoding of past experience in ventromedial prefrontal cortex

More than a century of research shows that spaced learning improves long-term memory. Yet, there remains debate concerning why. A major limitation to resolving theoretical debates is the lack of evidence for how neural representations change as a function of spacing. Here, leveraging a massive-scale 7T human fMRI dataset, we tracked neural representations and behavioral expressions of memory as participants viewed thousands of natural scene images that repeated at lags ranging from seconds to many months. We show that spaced learning increases the similarity of human ventromedial prefrontal cortex representations across stimulus encounters and, critically, these increases parallel and predict the behavioral benefits of spacing. Additionally, we show that these spacing benefits critically depend on remembering and, in turn, 're-encoding' past experience. Collectively, our findings provide fundamental insight into how spaced learning influences neural representations and why spacing is beneficial.

The persistence of memory: prior memory responses modulate behavior and brain state engagement

Memory brain states may influence how we experience an event. Memory encoding and retrieval constitute neurally dissociable brain states that individuals can selectively engage based on top-down goals. To the extent that memory states linger in time - as suggested by prior behavioral work - memory states may influence not only the current experience, but also subsequent stimuli and judgments. Thus lingering memory states may have broad influences on cognition, yet this account has not been directly tested utilizing neural measures of memory states. Here we address this gap by testing the hypothesis that memory brain states are modulated by memory judgments, and that these brain states persist for several hundred milliseconds. We recorded scalp electroencephalography (EEG) while participants completed a recognition memory task. We used an independently validated multivariate mnemonic state classifier to assess memory state engagement. We replicate prior behavioral findings; however, our neural findings run counter to the predictions made on the basis of the behavioral data. Surprisingly, we find that prior responses modulate current memory state engagement on the basis of response congruency. That is, we find strong engagement of the retrieval state on incongruent trials - when a target is preceded by a correct rejection of a lure and when a lure is preceded by successful recognition of a target. These findings indicate that cortical brain states are influenced by prior judgments and suggest that a non-mnemonic, internal attention state may be recruited to in the face of changing demands in a dynamic environment.

Top-down task goals induce the retrieval state

Engaging the retrieval state (Tulving, 1983) impacts processing and behavior (Long & Kuhl, 2019, 2021; Smith, Moore, & Long, 2022), but the extent to which top-down factors - explicit instructions and goals - vs. bottom-up factors - stimulus properties such as repetition and similarity - jointly or independently induce the retrieval state is unclear. Identifying the impact of bottom-up and top-down factors on retrieval state engagement is critical for understanding how control of task-relevant vs. task-irrelevant brain states influence cognition. We conducted between-subjects recognition memory tasks on male and female human participants in which we varied test phase goals. We recorded scalp electroencephalography and used an independently validated mnemonic state classifier (Long, 2023) to measure retrieval state engagement as a function of top-down task goals (recognize old vs. detect new items) and bottom-up stimulus repetition (hits vs. correct rejections). We find that whereas the retrieval state is engaged for hits regardless of top-down goals, the retrieval state is only engaged during correct rejections when the top-down goal is to recognize old items. Furthermore, retrieval state engagement is greater for low compared to high confidence hits when the task goal is to recognize old items. Together, these results suggest that top-down demands to recognize old items induce the retrieval state independent from bottom-up factors, potentially reflecting the recruitment of internal attention to enable access of a stored representation.

Significance Statement

Both top-down goals and automatic bottom-up influences may lead us into a retrieval brain state - a whole-brain pattern of activity that supports our ability to remember the past. Here we tested the extent to which top-down vs. bottom-up factors independently influence the retrieval state by manipulating participants' goals and stimulus repetition during a memory test. We find that in response to the top-down goal to recognize old items, the retrieval state is engaged for both old and new probes, suggesting that top-down and bottom-up factors independently engage the retrieval state. Our interpretation is that top-down demands recruit internal attention in service of the attempt to access a stored representation.

Multiple Memory Subsystems: Reconsidering Memory in the Mind and Brain

The multiple-memory-systems framework-that distinct types of memory are supported by distinct brain systems-has guided learning and memory research for decades. However, recent work challenges the one-to-one mapping between brain structures and memory types central to this taxonomy, with key memory-related structures supporting multiple functions across substructures. Here we integrate cross-species findings in the hippocampus, striatum, and amygdala to propose an updated framework of multiple memory subsystems (MMSS). We provide evidence for two organizational principles of the MMSS theory: First, opposing memory representations are colocated in the same brain structures; second, parallel memory representations are supported by distinct structures. We discuss why this burgeoning framework has the potential to provide a useful revision of classic theories of long-term memory, what evidence is needed to further validate the framework, and how this novel perspective on memory organization may guide future research.

Sensory processing in humans and mice fluctuates between external and internal modes

Perception is known to cycle through periods of enhanced and reduced sensitivity to external information. Here, we asked whether such slow fluctuations arise as a noise-related epiphenomenon of limited processing capacity or, alternatively, represent a structured mechanism of perceptual inference. Using 2 large-scale datasets, we found that humans and mice alternate between externally and internally oriented modes of sensory analysis. During external mode, perception aligns more closely with the external sensory information, whereas internal mode is characterized by enhanced biases toward perceptual history. Computational modeling indicated that dynamic changes in mode are enabled by 2 interlinked factors: (i) the integration of subsequent inputs over time and (ii) slow antiphase oscillations in the impact of external sensory information versus internal predictions that are provided by perceptual history. We propose that between-mode fluctuations generate unambiguous error signals that enable optimal inference in volatile environments.

Time for Memories

The ability to store information about the past to dynamically predict and prepare for the future is among the most fundamental tasks the brain performs. To date, the problems of understanding how the brain stores and organizes information about the past (memory) and how the brain represents and processes temporal information for adaptive behavior have generally been studied as distinct cognitive functions. This Symposium explores the inherent link between memory and temporal cognition, as well as the potential shared neural mechanisms between them. We suggest that working memory and implicit timing are interconnected and may share overlapping neural mechanisms. Additionally, we explore how temporal structure is encoded in associative and episodic memory and, conversely, the influences of episodic memory on subsequent temporal anticipation and the perception of time. We suggest that neural sequences provide a general computational motif that contributes to timing and working memory, as well as the spatiotemporal coding and recall of episodes.

Anterior Hippocampal Engagement during Memory Formation Predicts Subsequent False Recognition of Similar Experiences

People better remember experiences when they orient to meaning over surface-level perceptual features. Such an orientation-related memory boost has been associated with engagement of both hippocampus (HPC) and neocortex during encoding. However, less is known about the neural mechanisms by which a cognitive orientation toward meaning might also promote memory errors, with one open question being whether the HPC-a region traditionally implicated in precise memory formation-also contributes to behavioral imprecision. We used fMRI to characterize encoding-phase signatures as people oriented toward the meaning (story) versus perceptual style (artist) of storybook-style illustrations and then linked them to subsequent true and false memories. We found that story and artist orientation tasks were each associated with both unique univariate profiles and consistent neural states defined using multivoxel patterns. Linking these neural signatures to behavior, we found that greater medial pFC activation and alignment of neural patterns to the story (but not artist) state was related to subsequent memory success on a trial-by-trial basis. Moreover, among successfully remembered experiences, greater anterior HPC engagement at encoding was associated with a higher likelihood of related false recognitions, consistent with the encoding of broad traces in this region. Interestingly, these effects did not reliably vary by cued orientation. These results suggest that, irrespective of the cued encoding orientation, neocortical and hippocampal mechanisms associated with orienting to meaning (story) over perceptual (artist) features may support memory, with the formation of generalizable memories being a specialty of anterior HPC.

Conceptual relatedness promotes memory generalization at the cost of detailed recollection

An adaptive memory system is one that allows us to both retrieve detailed memories as well as generalize knowledge about our past, the latter termed memory generalization and is useful for making inferences about new situations. Research has indicated that memory generalization relies on forming knowledge structures by integrating experiences with shared encountered elements. Whether memory generalization occurs more readily when  experiences also have elements that share established (conceptual) information is less clear. It is also unclear if engaging in memory generalization during learning comes at the cost of retrieving detailed memories, the other function of episodic memory. To address these two knowledge gaps, we paired a modified version of the acquired equivalence task with a recognition memory test. Across three experiments, participants first learned a series of overlapping object-scene pairs (A-X, B-X and A-Y) in which half of the overlapping pairs contained conceptually-related objects (e.g., A-pencil; B-scissors; conceptual condition) and the other half contained unrelated objects (neutral condition). Participants ability to generalize to new overlapping object-scene pairs (B-Y) as well as not-learned but semantically-related objects was measured. Finally, participants completed a recognition memory test that included the encoded objects, perceptually similar lures or new foil objects. Across all experiments, we found higher rates of generalization but reduced detailed memory (indexed by increased false alarms to lure objects) for information learned in the conceptual than neutral condition. These results suggest the presence of conceptual knowledge biases an individual towards a generalization function of memory, which comes at the expense of detailed recollection.

Spatiotemporal Dynamics of Memory Encoding and Memory Retrieval States

Memory encoding and memory retrieval are neurally distinct brain states that can be differentiated on the basis of cortical network activity. However, it is unclear whether sustained engagement of one network or fluctuations between multiple networks give rise to these memory states. The spatiotemporal dynamics of memory states may have important implications for memory behavior and cognition; however, measuring temporally resolved signals of cortical networks poses a challenge. Here, we recorded scalp electroencephalography from participants performing a mnemonic state task in which they were biased toward memory encoding or retrieval. We performed a microstate analysis to measure the temporal dynamics of cortical networks throughout this mnemonic state task. We find that Microstate E, a putative analog of the default mode network, shows temporally sustained dissociations between memory encoding and retrieval, with greater engagement during retrieve compared with encode trials. We further show that decreased engagement of Microstate E is a general property of encoding, rather than a reflection of retrieval suppression. Thus, memory success, as well as cognition more broadly, may be influenced by the ability to engage or disengage Microstate E in a goal-dependent manner.

Perception and memory retrieval states are reflected in distributed patterns of background functional connectivity

The same visual input can serve as the target of perception or as a trigger for memory retrieval depending on whether cognitive processing is externally oriented (perception) or internally oriented (memory retrieval). While numerous human neuroimaging studies have characterized how visual stimuli are differentially processed during perception versus memory retrieval, perception and memory retrieval may also be associated with distinct neural states that are independent of stimulus-evoked neural activity. Here, we combined human fMRI with full correlation matrix analysis (FCMA) to reveal potential differences in "background" functional connectivity across perception and memory retrieval states. We found that perception and retrieval states could be discriminated with high accuracy based on patterns of connectivity across (1) the control network, (2) the default mode network (DMN), and (3) retrosplenial cortex (RSC). In particular, clusters in the control network increased connectivity with each other during the perception state, whereas clusters in the DMN were more strongly coupled during the retrieval state. Interestingly, RSC switched its coupling between networks as the cognitive state shifted from retrieval to perception. Finally, we show that background connectivity (1) was fully independent from stimulus-related variance in the signal and, further, (2) captured distinct aspects of cognitive states compared to traditional classification of stimulus-evoked responses. Together, our results reveal that perception and memory retrieval are associated with sustained cognitive states that manifest as distinct patterns of connectivity among large-scale brain networks.

Temporal Dynamics of Competition between Statistical Learning and Episodic Memory in Intracranial Recordings of Human Visual Cortex

The function of long-term memory is not just to reminisce about the past, but also to make predictions that help us behave appropriately and efficiently in the future. This predictive function of memory provides a new perspective on the classic question from memory research of why we remember some things but not others. If prediction is a key outcome of memory, then the extent to which an item generates a prediction signifies that this information already exists in memory and need not be encoded. We tested this principle using human intracranial EEG as a time-resolved method to quantify prediction in visual cortex during a statistical learning task and link the strength of these predictions to subsequent episodic memory behavior. Epilepsy patients of both sexes viewed rapid streams of scenes, some of which contained regularities that allowed the category of the next scene to be predicted. We verified that statistical learning occurred using neural frequency tagging and measured category prediction with multivariate pattern analysis. Although neural prediction was robust overall, this was driven entirely by predictive items that were subsequently forgotten. Such interference provides a mechanism by which prediction can regulate memory formation to prioritize encoding of information that could help learn new predictive relationships.SIGNIFICANCE STATEMENT When faced with a new experience, we are rarely at a loss for what to do. Rather, because many aspects of the world are stable over time, we rely on past experiences to generate expectations that guide behavior. Here we show that these expectations during a new experience come at the expense of memory for that experience. From intracranial recordings of visual cortex, we decoded what humans expected to see next in a series of photographs based on patterns of neural activity. Photographs that generated strong neural expectations were more likely to be forgotten in a later behavioral memory test. Prioritizing the storage of experiences that currently lead to weak expectations could help improve these expectations in future encounters.

Hippocampal convergence during anticipatory midbrain activation promotes subsequent memory formation

The hippocampus has been a focus of memory research since H.M's surgery abolished his ability to form new memories, yet its mechanistic role in memory remains debated. Here, we identify a candidate memory mechanism: an anticipatory hippocampal "convergence state", observed while awaiting valuable information, and which predicts subsequent learning. During fMRI, participants viewed trivia questions eliciting high or low curiosity, followed seconds later by its answer. We reasoned that encoding success requires a confluence of conditions, so that hippocampal states more conducive to memory formation should converge in state space. To operationalize convergence of neural states, we quantified the typicality of multivoxel patterns in the medial temporal lobes during anticipation and encoding of trivia answers. We found that the typicality of anticipatory hippocampal patterns increased during high curiosity. Crucially, anticipatory hippocampal pattern typicality increased with dopaminergic midbrain activation and uniquely accounted for the association between midbrain activation and subsequent recall. We propose that hippocampal convergence states may complete a cascade from motivation and midbrain activation to memory enhancement, and may be a general predictor of memory formation.

Tag and capture: how salient experiences target and rescue nearby events in memory

The long-term fate of a memory is not exclusively determined by the events occurring at the moment of encoding. Research at the cellular, circuit, and behavioral levels is beginning to reveal how neurochemical activations in the moments surrounding an event can retroactively and proactively rescue weak memory for seemingly mundane experiences. We review emerging evidence showing enhancement of weakly formed memories encoded minutes to hours before or after a related motivationally relevant experience. We discuss proposed neurobiological mechanisms for strengthening weak memories formed in temporal proximity to a strong event, and how this knowledge could be leveraged to improve memory for information that is prone to forgetting.

Experiencing Surprise: The Temporal Dynamics of Its Impact on Memory

To efficiently process information, the brain shifts between encoding and retrieval states, prioritizing bottom-up or top-down processing accordingly. Expectation violation before or during learning has been shown to trigger an adaptive encoding mechanism, resulting in better memory for unexpected events. Using fMRI, we explored (1) whether this encoding mechanism is also triggered during retrieval, and if so, (2) what the temporal dynamics of its mnemonic consequences are. Male and female participants studied object images, then, with new objects, they learned a contingency between a cue and a semantic category. Rule-abiding (expected) and violating (unexpected) targets and similar foils were used at test. We found interactions between previous and current similar events' expectation, such that when an expected event followed a similar but unexpected event, its performance was boosted, underpinned by activation in the hippocampus, midbrain, and occipital cortex. In contrast, a sequence of two unexpected similar events also triggered occipital engagement; however, this did not enhance memory performance. Taken together, our findings suggest that when the goal is to retrieve, encountering surprising events engages an encoding mechanism, supported by bottom-up processing, that may enhance memory for future related events.SIGNIFICANCE STATEMENT Optimizing the balance between new learning and the retrieval of existing knowledge is an ongoing process, at the core of human cognition. Previous research into memory encoding suggests experiencing surprise leads to the prioritization of the learning of new memories, forming an adaptive encoding mechanism. We examined whether this mechanism is also engaged when the current goal is to retrieve information. Our results demonstrate that an expectation-driven shift toward an encoding state, supported by enhanced perceptual processing, is beneficial for the correct identification of subsequent expected similar events. These findings have important implications for our understanding of the temporal dynamics of the adaptive encoding of information into memory.

Predicting memory from the network structure of naturalistic events

When we remember events, we often do not only recall individual events, but also the connections between them. However, extant research has focused on how humans segment and remember discrete events from continuous input, with far less attention given to how the structure of connections between events impacts memory. Here we conduct a functional magnetic resonance imaging study in which participants watch and recall a series of realistic audiovisual narratives. By transforming narratives into networks of events, we demonstrate that more central events-those with stronger semantic or causal connections to other events-are better remembered. During encoding, central events evoke larger hippocampal event boundary responses associated with memory formation. During recall, high centrality is associated with stronger activation in cortical areas involved in episodic recollection, and more similar neural representations across individuals. Together, these results suggest that when humans encode and retrieve complex real-world experiences, the reliability and accessibility of memory representations is shaped by their location within a network of events.

Hippocampal representations switch from errors to predictions during acquisition of predictive associations

We constantly exploit the statistical regularities in our environment to help guide our perception. The hippocampus has been suggested to play a pivotal role in both learning environmental statistics, as well as exploiting them to generate perceptual predictions. However, it is unclear how the hippocampus balances encoding new predictive associations with the retrieval of existing ones. Here, we present the results of two high resolution human fMRI studies (N = 24 for both experiments) directly investigating this. Participants were exposed to auditory cues that predicted the identity of an upcoming visual shape (with 75% validity). Using multivoxel decoding analysis, we find that the hippocampus initially preferentially represents unexpected shapes (i.e., those that violate the cue regularities), but later switches to representing the cue-predicted shape regardless of which was actually presented. These findings demonstrate that the hippocampus is involved both acquiring and exploiting predictive associations, and is dominated by either errors or predictions depending on whether learning is ongoing or complete.

A neural network model of when to retrieve and encode episodic memories

Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.

Uncertainty alters the balance between incremental learning and episodic memory

A key question in decision-making is how humans arbitrate between competing learning and memory systems to maximize reward. We address this question by probing the balance between the effects, on choice, of incremental trial-and-error learning versus episodic memories of individual events. Although a rich literature has studied incremental learning in isolation, the role of episodic memory in decision-making has only recently drawn focus, and little research disentangles their separate contributions. We hypothesized that the brain arbitrates rationally between these two systems, relying on each in circumstances to which it is most suited, as indicated by uncertainty. We tested this hypothesis by directly contrasting contributions of episodic and incremental influence to decisions, while manipulating the relative uncertainty of incremental learning using a well-established manipulation of reward volatility. Across two large, independent samples of young adults, participants traded these influences off rationally, depending more on episodic information when incremental summaries were more uncertain. These results support the proposal that the brain optimizes the balance between different forms of learning and memory according to their relative uncertainties and elucidate the circumstances under which episodic memory informs decisions.

Mixing memory and desire: How memory reactivation supports deliberative decision-making

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affects deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted based on their uncertainty. We close by reviewing related findings and identifying areas for further research. This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition Neuroscience > Computation.

Examining the transition of novel information toward familiarity

Throughout their lives, humans encounter multiple instances of new information that can be inconsistent with prior knowledge (novel). Over time, the once-novel information becomes integrated into their established knowledge base, shifting from novelty to familiarity. In this study, we investigated the processes by which the first steps of this transition take place. We hypothesized that the neural representations of initially novel items gradually change over the course of repeated presentations, expressing a shift toward familiarity. We further assumed that this shift could be traced by examining neural patterns using fMRI. In two experiments, while being scanned, participants read noun-adjective word pairs that were either consistent or inconsistent with their prior knowledge. Stimuli were repeated 3-6 times within the scans. Employing mass univariate and multivariate similarity analyses, we showed that the neural representations associated with the initial presentation of familiar versus novel objects differed in lateral frontal and temporal regions, the medial prefrontal cortex, and the medial temporal lobe. Importantly, the neural representations of novel stimuli gradually changed throughout repetitions until they became indistinguishable from their respective familiar items. We interpret these findings as indicating that an early phase of familiarization can be completed within a few repetitions. This initial familiarization can then serve as the prerequisite to the integration of novel items into existing knowledge. Future empirical and theoretical works can build on the current findings to develop a comprehensive model of the transition from novelty to familiarity.

Generalization and False Memory in an Acquired Equivalence Paradigm: The Influence of Physical Resemblance Across Related Episodes

The ability to make inferences about related experiences is an important function of memory that allows individuals to build generalizable knowledge. In some cases, however, making inferences may lead to false memories when individuals misremember inferred information as having been observed. One factor that is known to increase the prevalence of false memories is the physical resemblance between new and old information. The extent to which physical resemblance has parallel effects on generalization and memory for the source of inferred associations is not known. To investigate the parallels between memory generalization and false memories, we conducted three experiments using an acquired equivalence paradigm and manipulated physical resemblance between items that made up related experiences. The three experiments showed increased generalization for higher levels of resemblance. Recognition and source memory judgments revealed that high rates of generalization were not always accompanied by high rates of false memories. Thus, physical resemblance across episodes may promote generalization with or without a trade-off in terms of impeding memory specificity.

Cross-Cultural Differences in Memory Specificity: Investigation of Candidate Mechanisms

Previous research has revealed that people from Western cultures tend to remember more details of objects and events in autobiographical memory compared to people from Eastern cultures. The present experiments tested whether differences in pattern separation - the process by which new, but potentially similar, exemplars are discriminated from previously-encountered exemplars - account for these cultural difference in object memory. In two experiments, we investigated the extent to which North Americans and East Asians differ in pattern separation and whether these effects are related to cultural values. We also examined the role of response bias. These results revealed it is unlikely that pattern separation is the sole mechanism underlying cross-cultural memory specificity differences, as broader memory mechanisms, such as differences in memory resolution for previously-encoded items, could account for the differences observed between groups.

Bistable perception alternates between internal and external modes of sensory processing

Perceptual history can exert pronounced effects on the contents of conscious experience: when confronted with completely ambiguous stimuli, perception does not waver at random between diverging stimulus interpretations but sticks with recent percepts for prolonged intervals. Here, we investigated the relevance of perceptual history in situations more similar to everyday experience, where sensory stimuli are usually not completely ambiguous. Using partially ambiguous visual stimuli, we found that the balance between past and present is not stable over time but slowly fluctuates between two opposing modes. For time periods of up to several minutes, perception was either largely determined by perceptual history or driven predominantly by disambiguating sensory evidence. Computational modeling suggested that the construction of unambiguous conscious experiences is modulated by slow fluctuations between internally and externally oriented modes of sensory processing.

Signed and unsigned reward prediction errors dynamically enhance learning and memory

Memory helps guide behavior, but which experiences from the past are prioritized? Classic models of learning posit that events associated with unpredictable outcomes as well as, paradoxically, predictable outcomes, deploy more attention and learning for those events. Here, we test reinforcement learning and subsequent memory for those events, and treat signed and unsigned reward prediction errors (RPEs), experienced at the reward-predictive cue or reward outcome, as drivers of these two seemingly contradictory signals. By fitting reinforcement learning models to behavior, we find that both RPEs contribute to learning by modulating a dynamically changing learning rate. We further characterize the effects of these RPE signals on memory and show that both signed and unsigned RPEs enhance memory, in line with midbrain dopamine and locus-coeruleus modulation of hippocampal plasticity, thereby reconciling separate findings in the literature.

Temporally coherent perturbation of neural dynamics during retention alters human multi-item working memory

Temporarily storing a list of items in working memory (WM), a fundamental ability in cognition, has been posited to rely on the temporal dynamics of multi-item neural representations during retention. However, the causal evidence, particularly in human subjects, is still lacking, let alone WM manipulation. Here, we develop a novel "dynamic perturbation" approach to manipulate the relative memory strength of WM items held in human brain, by presenting temporally correlated luminance sequences during retention to interfere with the multi-item neural dynamics. Six experiments on more than 150 subjects confirm the effectiveness of this WM manipulation approach. A computational model combining continuous attractor neural network (CANN) and short-term synaptic plasticity (STP) principles further reproduces all the empirical findings. The model reveals that the "dynamic perturbation" modifies the synaptic efficacies of WM items through STP principles, eventually leading to changes in their relative memory strengths. Our results support the causal role of temporal dynamics of neural network in mediating multi-item WM, and offer a promising, purely bottom-up approach to manipulate WM.

Cortical Representations of Visual Stimuli Shift Locations with Changes in Memory States

Episodic memory retrieval is thought to rely on reactivation of the same content-sensitive neural activity patterns initially expressed during memory encoding.^1-6 Yet there are emerging examples of content representations expressed in different brain regions during encoding versus retrieval.^7-14 Although these differences have been observed by comparing encoding and retrieval tasks that differ in terms of perceptual experience and cognitive demands, there are many real-world contexts-e.g., meeting a new colleague who reminds you of an old acquaintance-where the memory system might be intrinsically biased either toward encoding (the new colleague) or retrieval (the old acquaintance).¹⁵¹⁶ Here, we test whether intrinsic memory states, independent of task demands, determine the cortical location of content representations. In a human fMRI study, subjects (n = 33) viewed object images and were instructed to either encode the current object or retrieve a similar object from memory. Using pattern classifiers, we show that biases toward encoding versus retrieval were reflected in large-scale attentional networks.^17-19 Critically, memory states decoded from these networks-even when entirely independent from task instructions-predicted shifts of object representations from visual cortex (encoding) to ventral parietal cortex (retrieval). Finally, visual versus ventral parietal cortices exhibited differential connectivity with the hippocampus during memory encoding versus retrieval, consistent with the idea that the hippocampus mediates cortical shifts in content representations. Collectively, these findings demonstrate that intrinsic biases toward memory encoding versus retrieval determine the specific cortical locations that express content information.

Statistical learning as a reference point for memory distortions: Swap and shift errors

Humans use regularities in the environment to facilitate learning, often without awareness or intent. How might such regularities distort long-term memory? Here, participants studied and reported the colors of objects in a long-term memory paradigm, uninformed that certain colors were sampled more frequently overall. When participants misreported an object's color, these errors were often centered around the average studied color (i.e., "Rich" color), demonstrating swap errors in long-term memory due to imposed statistical regularities. We observed such swap errors regardless of memory load, explicit knowledge, or the distance in color space between the correct color of the tested object and the Rich color. An explicit guessing strategy where participants intentionally made swap errors when uncertain could not fully account for our results. We discuss other potential sources of observed swap errors such as false memory and implicit biased guessing. Although less robust than swap errors, evidence was also observed for subtle shift errors towards or away from the Rich color dependent on the color distance between the correct color and the Rich color. Together, these findings of swap and shift errors provide converging evidence for memory distortion mechanisms induced by a reference point, bridging a gap in the literature between how attention to regularities similarly influences visual working memory and visual long-term memory.

Acetylcholine and Spontaneous Recognition Memory in Rodents and Primates

Whilst acetylcholine has long been linked to memory, there have been significant questions about its specific role. In particular, the effects of cholinergic manipulations in primates and rodents has often been at odds. Here, we review the work in primates and rodents on the specific function of acetylcholine in memory, and episodic memory in particular. We propose that patterns of impairment can best be understood in terms of a role for hippocampal acetylcholine in resolving spatial interference and we discuss the benefits of new tasks of episodic memory in animals allowing clearer translation of findings to the clinic.

When the Memory System Gets Ahead of Itself

Humans are adept at learning and exploiting statistical regularities to predict future events from current experience. A recent paper by Sherman and Turk-Browne demonstrates that statistical regularities bias the hippocampus toward representing future states over current experience and reduce the degree to which current experience is encoded into memory.

Prior knowledge promotes hippocampal separation but cortical assimilation in the left inferior frontal gyrus

An adaptive memory system rarely learns information tabula rasa, but rather builds on prior knowledge to facilitate learning. How prior knowledge influences the neural representation of novel associations remains unknown. Here, participants associated pairs of faces in two conditions: a famous, highly familiar face with a novel face or two novel faces while undergoing fMRI. We examine multivoxel activity patterns corresponding to individual faces before and after learning. The activity patterns representing members of famous-novel pairs becomes separated in the hippocampus, that is, more distinct from one another through learning, in striking contrast to paired novel faces that become similar. In the left inferior frontal gyrus, however, prior knowledge leads to integration, and in a specific direction: the representation of the novel face becomes similar to that of the famous face after learning, suggesting assimilation of new into old memories. We propose that hippocampal separation might resolve interference between existing and newly learned information, allowing cortical assimilation. Thus, associative learning with versus without prior knowledge relies on radically different computations.

Prior knowledge strongly impacts new learning, but its influence on the neural representation of novel information is unknown. Here, the authors show multiple neural codes for learning: prior knowledge leads to integrated cortical representations, while promoting hippocampal separation.

Statistical prediction of the future impairs episodic encoding of the present

Memory is typically thought of as enabling reminiscence about past experiences. However, memory also informs and guides processing of future experiences. These two functions of memory are often at odds: Remembering specific experiences from the past requires storing idiosyncratic properties that define particular moments in space and time, but by definition such properties will not be shared with similar situations in the future and thus may not be applicable to future situations. We discovered that, when faced with this conflict, the brain prioritizes prediction over encoding. Behavioral tests of recognition and source recall showed that items allowing for prediction of what will appear next based on learned regularities were less likely to be encoded into memory. Brain imaging revealed that the hippocampus was responsible for this interference between statistical learning and episodic memory. The more that the hippocampus predicted the category of an upcoming item, the worse the current item was encoded. This competition may serve an adaptive purpose, focusing encoding on experiences for which we do not yet have a predictive model.

Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully

Large-scale collaborative projects recently demonstrated that several key findings from the social-science literature could not be replicated successfully. Here, we assess the extent to which a finding’s replication success relates to its intuitive plausibility. Each of 27 high-profile social-science findings was evaluated by 233 people without a Ph.D. in psychology. Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%). In addition, when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67%. We discuss the prediction patterns and apply signal detection theory to disentangle detection ability from response bias. Our study suggests that laypeople’s predictions contain useful information for assessing the probability that a given finding will be replicated successfully.

Mnemonic prediction errors bias hippocampal states

When our experience violates our predictions, it is adaptive to upregulate encoding of novel information, while down-weighting retrieval of erroneous memory predictions to promote an updated representation of the world. We asked whether mnemonic prediction errors promote hippocampal encoding versus retrieval states, as marked by distinct network connectivity between hippocampal subfields. During fMRI scanning, participants were cued to internally retrieve well-learned complex room-images and were then presented with either an identical or a modified image (0-4 changes). In the left hemisphere, we find that CA1-entorhinal connectivity increases, and CA1-CA3 connectivity decreases, with the number of changes. Further, in the left CA1, the similarity between activity patterns during cued-retrieval of the learned room and during the image is lower when the image includes changes, consistent with a prediction error signal in CA1. Our findings provide a mechanism by which mnemonic prediction errors may drive memory updating—by biasing hippocampal states.

When our expectations are violated, it is adaptive to update our internal models to improve predictions in the future. Here, the authors show that during mnemonic violations, hippocampal networks are biased towards an encoding state and away from a retrieval state to potentially update these predictions.

Reward prediction errors create event boundaries in memory

We remember when things change. Particularly salient are experiences where there is a change in rewards, eliciting reward prediction errors (RPEs). How do RPEs influence our memory of those experiences? One idea is that this signal directly enhances the encoding of memory. Another, not mutually exclusive, idea is that the RPE signals a deeper change in the environment, leading to the mnemonic separation of subsequent experiences from what came before, thereby creating a new latent context and a more separate memory trace. We tested this in four experiments where participants learned to predict rewards associated with a series of trial-unique images. High-magnitude RPEs indicated a change in the underlying distribution of rewards. To test whether these large RPEs created a new latent context, we first assessed recognition priming for sequential pairs that included a high-RPE event or not (Exp. 1: n = 27 & Exp. 2: n = 83). We found evidence of recognition priming for the high-RPE event, indicating that the high-RPE event is bound to its predecessor in memory. Given that high-RPE events are themselves preferentially remembered (Rouhani, Norman, & Niv, 2018), we next tested whether there was an event boundary across a high-RPE event (i.e., excluding the high-RPE event itself; Exp. 3: n = 85). Here, sequential pairs across a high RPE no longer showed recognition priming whereas pairs within the same latent reward state did, providing initial evidence for an RPE-modulated event boundary. We then investigated whether RPE event boundaries disrupt temporal memory by asking participants to order and estimate the distance between two events that had either included a high-RPE event between them or not (Exp. 4). We found (n = 49) and replicated (n = 77) worse sequence memory for events across a high RPE. In line with our recognition priming results, we did not find sequence memory to be impaired between the high-RPE event and its predecessor, but instead found worse sequence memory for pairs across a high-RPE event. Moreover, greater distance between events at encoding led to better sequence memory for events across a low-RPE event, but not a high-RPE event, suggesting separate mechanisms for the temporal ordering of events within versus across a latent reward context. Altogether, these findings demonstrate that high-RPE events are both more strongly encoded, show intact links with their predecessor, and act as event boundaries that interrupt the sequential integration of events. We captured these effects in a variant of the Context Maintenance and Retrieval model (CMR; Polyn, Norman, & Kahana, 2009), modified to incorporate RPEs into the encoding process.

Interference between overlapping memories is predicted by neural states during learning

One of the primary contributors to forgetting is interference from overlapping memories. Intuitively, this suggests-and prominent theoretical models argue-that memory interference is best avoided by encoding overlapping memories as if they were unrelated. It is therefore surprising that reactivation of older memories during new encoding has been associated with reduced memory interference. Critically, however, prior studies have not directly established why reactivation reduces interference. Here, we first developed a behavioral paradigm that isolates the negative influence that overlapping memories exert during memory retrieval. We then show that reactivating older memories during the encoding of new memories dramatically reduces this interference cost at retrieval. Finally, leveraging multiple fMRI decoding approaches, we show that spontaneous reactivation of older memories during new encoding leads to integration of overlapping memories and, critically, that integration during encoding specifically reduces interference between overlapping, and otherwise competing, memories during retrieval.

The hippocampus as a visual area organized by space and time: A spatiotemporal similarity hypothesis

The hippocampus is the canonical memory system in the brain and is not typically considered part of the visual system. Yet, it sits atop the ventral visual stream and has been implicated in certain aspects of vision. Here I review the place of the hippocampal memory system in vision science. After a brief primer on the local circuity, external connectivity, and computational functions of the hippocampus, I explore what can be learned from each field about the other. I first present four areas of vision science (scene perception, imagery, eye movements, attention) that challenge our current understanding of the hippocampus in terms of its role in episodic memory. In the reverse direction, I leverage this understanding to inform vision science in other ways, presenting a working hypothesis about a unique form of visual representation. This spatiotemporal similarity hypothesis states that the hippocampus represents objects according to whether they co-occur in space and/or time, and not whether they look alike, as elsewhere in the visual system. This tuning may reflect hippocampal mechanisms of pattern separation, relational binding, and statistical learning, allowing the hippocampus to generate visual expectations to facilitate search and recognition.

Modulating the Use of Multiple Memory Systems in Value-based Decisions with Contextual Novelty

With multiple learning and memory systems at its disposal, the human brain can represent the past in many ways, from extracting regularities across similar experiences (incremental learning) to storing rich, idiosyncratic details of individual events (episodic memory). The unique information carried by these neurologically distinct forms of memory can bias our behavior in different directions, raising crucial questions about how these memory systems interact to guide choice and the factors that cause one to dominate. Here, we devised a new approach to estimate how decisions are independently influenced by episodic memories and incremental learning. Furthermore, we identified a biologically motivated factor that biases the use of different memory types-the detection of novelty versus familiarity. Consistent with computational models of cholinergic memory modulation, we find that choices are more influenced by episodic memories following the recognition of an unrelated familiar image but more influenced by incrementally learned values after the detection of a novel image. Together this work provides a new behavioral tool enabling the disambiguation of key memory behaviors thought to be supported by distinct neural systems while also identifying a theoretically important and broadly applicable manipulation to bias the arbitration between these two sources of memories.

Decoding the tradeoff between encoding and retrieval to predict memory for overlapping events

When new events overlap with past events, there is a natural tradeoff between encoding the new event and retrieving the past event. Given the ubiquity of overlap among memories, this tradeoff between memory encoding and retrieval is of central importance to computational models of episodic memory (O'Reilly & McClelland 1994; Hasselmo 2005). However, prior studies have not directly linked neural markers of encoding/retrieval tradeoffs to behavioral measures of how overlapping events are remembered. Here, by decoding patterns of scalp electroencephalography (EEG) from male and female human subjects, we show that tradeoffs between encoding and retrieval states are reflected in distributed patterns of neural activity and, critically, these neural tradeoffs predict how overlapping events will later be remembered. Namely, new events that overlapped with past events were more likely to be subsequently remembered if neural patterns were biased toward a memory encoding state-or, conversely, away from a retrieval state. Additionally, we show that neural markers of encoding vs. retrieval states are surprisingly independent from previously-described EEG predictors of subsequent memory. Instead, we demonstrate that previously-described EEG predictors of subsequent memory are better explained by task engagement than by memory encoding, per se. Collectively, our findings provide important insight into how the memory system balances memory encoding and retrieval states and, more generally, into the neural mechanisms that support successful memory formation.

Seeing the past: afterglow effects on familiarity judgments are category-specific

According to several computational models, novel items can create a learning mode with dynamics favorable to new learning, and not to memory retrieval. In line with that idea, a new item in a recognition test has been found to create a bias toward calling subsequent items new as well. Here, we tested whether this bias, which we termed the afterglow effect, is indeed caused by a general learning mode, or is caused by perceptual overlap between preceding and current items. In two experiments, we show that a preceding recognition judgment biases the current one, but only if the preceding and current items are of the same perceptual category. In contrast, we did not find strong bias effects from perceptually novel fractal images, as would be predicted if novel items induce a learning mode that then biases recognition judgments. We conclude that the afterglow effect is more likely to reflect perceptual phenomena than a learning mode. We suggest how this can be reconciled with what is known about familiarity at the neural level.

Forgetting from lapses of sustained attention

When performing any task for an extended period of time, attention fluctuates between good and bad states. These fluctuations affect performance in the moment, but may also have lasting consequences for what gets encoded into memory. Experiment 1 establishes this relationship between attentional states and memory, by showing that subsequent memory for an item was predicted by a response time index of sustained attention (average response time during the three trials prior to stimulus onset). Experiment 2 strengthens the causal interpretation of this predictive relationship by treating the sustained attention index as an independent variable to trigger the appearance of an encoding trial. Subsequent memory was better when items were triggered from good versus bad attentional states. Together, these findings suggest that sustained attention can have downstream consequences for what we remember, and they highlight the inferential utility of adaptive experimental designs. By continuously monitoring attention, we can influence what will later be remembered.

Expected Value of Reward Predicts Episodic Memory for Incidentally Learnt Reward-Item Associations

In this paper, we draw connections between reward processing and cognition by behaviourally testing the implications of neurobiological theories of reward processing on memory. Single-cell neurophysiology in non-human primates and imaging work in humans suggests that the dopaminergic reward system responds to different components of reward: expected value; outcome or prediction error; and uncertainty of reward (Schultz et al., 2008). The literature on both incidental and motivated learning has focused on understanding how expected value and outcome—linked to increased activity in the reward system—lead to consolidation-related memory enhancements. In the current study, we additionally investigate the impact of reward uncertainty on human memory. The contribution of reward uncertainty—the spread of the reward probability distribution irrespective of the magnitude—has not been previously examined. To examine the effects of uncertainty on memory, a word-learning task was introduced, along with a surprise delayed recognition memory test. Using Bayesian model selection, we found evidence only for expected value as a predictor of memory performance. Our findings suggest that reward uncertainty does not enhance memory for individual items. This supports emerging evidence that an effect of uncertainty on memory is only observed in high compared to low risk environments.

Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Being able to replicate scientific findings is crucial for scientific progress^1-15. We replicate 21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015^16-36. The replications follow analysis plans reviewed by the original authors and pre-registered prior to the replications. The replications are high powered, with sample sizes on average about five times higher than in the original studies. We find a significant effect in the same direction as the original study for 13 (62%) studies, and the effect size of the replications is on average about 50% of the original effect size. Replicability varies between 12 (57%) and 14 (67%) studies for complementary replicability indicators. Consistent with these results, the estimated true-positive rate is 67% in a Bayesian analysis. The relative effect size of true positives is estimated to be 71%, suggesting that both false positives and inflated effect sizes of true positives contribute to imperfect reproducibility. Furthermore, we find that peer beliefs of replicability are strongly related to replicability, suggesting that the research community could predict which results would replicate and that failures to replicate were not the result of chance alone.

Time-resolved neural reinstatement and pattern separation during memory decisions in human hippocampus

Mnemonic decision-making has long been hypothesized to rely on hippocampal dynamics that bias memory processing toward the formation of new memories or the retrieval of old ones. Successful memory encoding may be best optimized by pattern separation, whereby two highly similar experiences can be represented by underlying neural populations in an orthogonal manner. By contrast, successful memory retrieval is thought to be supported by a recovery of the same neural pattern laid down during encoding. Here we examined how hippocampal pattern completion and separation emerge over time during memory decisions. We measured electrocorticography activity in the human hippocampus and posterior occipitotemporal cortex (OTC) while participants performed continuous recognition of items that were new, repeated (old), or highly similar to a prior item (similar). During retrieval decisions of old items, both regions exhibited significant reinstatement of multivariate high-frequency activity (HFA) associated with encoding. Further, the extent of reinstatement of encoding patterns during retrieval was correlated with the strength (HFA power) of hippocampal encoding. Evidence for encoding pattern reinstatement was also seen in OTC on trials requiring fine-grained discrimination of similar items. By contrast, hippocampal activity showed evidence for pattern separation during these trials. Together, these results underscore the critical role of the hippocampus in supporting both reinstatement of overlapping information and separation of similar events.