Expected Reward Value and Reward Uncertainty Have Temporally Dissociable Effects on Memory Formation

Contributions of transient and sustained reward to memory formation

Reward benefits to memory formation have been robustly linked to dopaminergic activity. Despite the established characterization of dopaminergic mechanisms as operating at multiple timescales, potentially supporting distinct functional outcomes, the temporal dynamics by which reward might modulate memory encoding are just beginning to be investigated. In the present study, we leveraged a mixed block/event experimental design to disentangle transient and sustained reward influences on task engagement and subsequent recognition memory in an adapted monetary-incentive-encoding (MIE) paradigm. Across three behavioral experiments, transient and sustained reward modulation of item and context memory was probed, at both 24-h and ~ 15-min retention intervals, to investigate the importance of overnight consolidation. In general, we observed that transient reward was associated with enhanced item memory encoding, while sustained reward modulated response speed but did not appear to benefit subsequent recognition accuracy. Notably, reward effects on item memory performance and response speed were somewhat inconsistent across the three experiments, with suggestions that RT speeding might also be related to time on task, and we did not observe reward modulation of context memory performance or amplification of reward benefits to memory by overnight consolidation. Taken together, the observed pattern of behavior is consistent with potentially distinct roles for transient and sustained reward in memory encoding and cognitive performance and suggests that further investigation of the temporal dynamics of dopaminergic contributions to memory formation will advance the understanding of motivated memory.

Signed and unsigned effects of prediction error on memory: Is it a matter of choice?

Adaptive decision-making is governed by at least two types of memory processes. On the one hand, learned predictions through integrating multiple experiences, and on the other hand, one-shot episodic memories. These two processes interact, and predictions - particularly prediction errors - influence how episodic memories are encoded. However, studies using computational models disagree on the exact shape of this relationship, with some findings showing an effect of signed prediction errors and others showing an effect of unsigned prediction errors on episodic memory. We argue that the choice-confirmation bias, which reflects stronger learning from choice-confirming compared to disconfirming outcomes, could explain these seemingly diverging results. Our perspective implies that the influence of prediction errors on episodic encoding critically depends on whether people can freely choose between options (i.e., instrumental learning tasks) or not (Pavlovian learning tasks). The choice-confirmation bias on memory encoding might have evolved to prioritize memory representations that optimize reward-guided decision-making. We conclude by discussing open issues and implications for future studies.

Multiple routes to enhanced memory for emotionally relevant events

Events associated with aversive or rewarding outcomes are prioritized in memory. This memory boost is commonly attributed to the elicited affective response, closely linked to noradrenergic and dopaminergic modulation of hippocampal plasticity. Herein we review and compare this 'affect' mechanism to an additional, recently discovered, 'prediction' mechanism whereby memories are strengthened by the extent to which outcomes deviate from expectations, that is, by prediction errors (PEs). The mnemonic impact of PEs is separate from the affective outcome itself and has a distinct neural signature. While both routes enhance memory, these mechanisms are linked to different - and sometimes opposing - predictions for memory integration. We discuss new findings that highlight mechanisms by which emotional events strengthen, integrate, and segment memory.

Reward expectation extinction restructures and degrades CA1 spatial maps through loss of a dopaminergic reward proximity signal

Hippocampal place cells support reward-related spatial memories by forming a cognitive map that over-represents reward locations. The strength of these memories is modulated by the extent of reward expectation during encoding. However, the circuit mechanisms underlying this modulation are unclear. Here we find that when reward expectation is extinguished in mice, they remain engaged with their environment, yet place cell over-representation of rewards vanishes, place field remapping throughout the environment increases, and place field trial-to-trial reliability decreases. Interestingly, Ventral Tegmental Area (VTA) dopaminergic axons in CA1 exhibit a ramping reward-proximity signal that depends on reward expectation and inhibiting VTA dopaminergic neurons largely replicates the effects of extinguishing reward expectation. We conclude that changing reward expectation restructures CA1 cognitive maps and determines map reliability by modulating the dopaminergic VTA-CA1 reward-proximity signal. Thus, internal states of high reward expectation enhance encoding of spatial memories by reinforcing hippocampal cognitive maps associated with reward.

Emotion dynamics as hierarchical Bayesian inference in time

What fundamental property of our environment would be most valuable and optimal in characterizing the emotional dynamics we experience in daily life? Empirical work has shown that an accurate estimation of uncertainty is necessary for our optimal perception, learning, and decision-making. However, the role of this uncertainty in governing our affective dynamics remains unexplored. Using Bayesian encoding, decoding and computational modeling, on a large-scale neuroimaging and behavioral data on a passive movie-watching task, we showed that emotions naturally arise due to ongoing uncertainty estimations about future outcomes in a hierarchical neural architecture. Several prefrontal subregions hierarchically encoded a lower-dimensional signal that highly correlated with the evolving uncertainty. Crucially, the lateral orbitofrontal cortex (lOFC) tracked the temporal fluctuations of this uncertainty and was predictive of the participants' predisposition to anxiety. Furthermore, we observed a distinct functional double-dissociation within OFC with increased connectivity between medial OFC and DMN, while with that of lOFC and FPN in response to the evolving affect. Finally, we uncovered a temporally predictive code updating an individual's beliefs spontaneously with fluctuating outcome uncertainty in the lOFC. A biologically relevant and computationally crucial parameter in the theories of brain function, we propose uncertainty to be central to the definition of complex emotions.

Reward at encoding but not retrieval modulates memory for detailed events

Much of the evidence suggesting that rewards improve memory performance has focused on how explicit rewards facilitate encoding of simplistic stimuli. To expand beyond this focus, the current study tested how explicit rewards presented at encoding as well as retrieval facilitate memory for information contained within complex events. In a single experimental session, participants (N = 88) encoded videos depicting naturalistic events (e.g., getting dressed) and then completed a recognition test probing their memory for different detail types (i.e., event, perceptual, or contextual) from the video stimuli. We manipulated the explicit reward associated with each video, such that accurate memory responses for half the videos were associated with high monetary incentives and half were associated with low monetary incentives. This reward manipulation was presented at either encoding or retrieval during a recognition memory test. The reward manipulation only affected memory when presented at encoding and this effect did not depend on the type of detail probed. Drift Diffusion Modelling further revealed that presenting reward information at encoding engendered greater encoding fidelity-indexed by an increase in drift rate-but did not change response caution at the time of retrieval-indexed by response threshold. Together, our results suggest that presenting reward information when encoding but not retrieving complex events has a general facilitatory effect, likely via attentional processing, on the ability to later remember precise details from the event.

Developmental change in prefrontal cortex recruitment supports the emergence of value-guided memory

Prioritizing memory for valuable information can promote adaptive behavior across the lifespan, but it is unclear how the neurocognitive mechanisms that enable the selective acquisition of useful knowledge develop. Here, using a novel task coupled with functional magnetic resonance imaging, we examined how children, adolescents, and adults (N = 90) learn from experience what information is likely to be rewarding, and modulate encoding and retrieval processes accordingly. We found that the ability to use learned value signals to selectively enhance memory for useful information strengthened throughout childhood and into adolescence. Encoding and retrieval of high- vs. low-value information was associated with increased activation in striatal and prefrontal regions implicated in value processing and cognitive control. Age-related increases in value-based lateral prefrontal cortex modulation mediated the relation between age and memory selectivity. Our findings demonstrate that developmental increases in the strategic engagement of the prefrontal cortex support the emergence of adaptive memory.

Anticipation of novel environments enhances memory for incidental information

Novelty is a potent driver of learning, but little is known about whether anticipation of novelty can enhance memory for incidental information. Here, participants incidentally encountered objects while they actively navigated toward novel or previously familiarized virtual rooms. Across immediate and delayed surprise memory tests, participants showed superior recollection for incidental objects encountered while anticipating novel as compared with familiarized rooms. Furthermore, memory for incidental objects correlated positively with between-participants average curiosity about novel rooms but negatively with within-participants trial-specific curiosity. Our findings contribute to the growing literature on how salient processes impact memory for incidental material.

Signed and unsigned reward prediction errors dynamically enhance learning and memory

Memory helps guide behavior, but which experiences from the past are prioritized? Classic models of learning posit that events associated with unpredictable outcomes as well as, paradoxically, predictable outcomes, deploy more attention and learning for those events. Here, we test reinforcement learning and subsequent memory for those events, and treat signed and unsigned reward prediction errors (RPEs), experienced at the reward-predictive cue or reward outcome, as drivers of these two seemingly contradictory signals. By fitting reinforcement learning models to behavior, we find that both RPEs contribute to learning by modulating a dynamically changing learning rate. We further characterize the effects of these RPE signals on memory and show that both signed and unsigned RPEs enhance memory, in line with midbrain dopamine and locus-coeruleus modulation of hippocampal plasticity, thereby reconciling separate findings in the literature.

Temporal proximity to the elicitation of curiosity is key for enhancing memory for incidental information

Curiosity states benefit memory for target information, but also incidental information presented during curiosity states. However, it is not known whether incidental curiosity-enhanced memory depends on when incidental information during curiosity states is encountered. Here, participants incidentally encoded unrelated face images at different time points while they anticipated answers to trivia questions. Across two experiments, we found memory enhancements for unrelated faces presented during high-curiosity compared with low-curiosity states, but only when presented shortly after a trivia question. This suggests processes associated with the elicitation of curiosity-but not sustained anticipation or the satisfaction of curiosity-enhance memory for incidental information.

Signed Reward Prediction Errors in the Ventral Striatum Drive Episodic Memory

Recent behavioral evidence implicates reward prediction errors (RPEs) as a key factor in the acquisition of episodic memory. Yet, important neural predictions related to the role of RPEs in episodic memory acquisition remain to be tested. Humans (both sexes) performed a novel variable-choice task where we experimentally manipulated RPEs and found support for key neural predictions with fMRI. Our results show that in line with previous behavioral observations, episodic memory accuracy increases with the magnitude of signed (i.e., better/worse-than-expected) RPEs (SRPEs). Neurally, we observe that SRPEs are encoded in the ventral striatum (VS). Crucially, we demonstrate through mediation analysis that activation in the VS mediates the experimental manipulation of SRPEs on episodic memory accuracy. In particular, SRPE-based responses in the VS (during learning) predict the strength of subsequent episodic memory (during recollection). Furthermore, functional connectivity between task-relevant processing areas (i.e., face-selective areas) and hippocampus and ventral striatum increased as a function of RPE value (during learning), suggesting a central role of these areas in episodic memory formation. Our results consolidate reinforcement learning theory and striatal RPEs as key factors subtending the formation of episodic memory.SIGNIFICANCE STATEMENT Recent behavioral research has shown that reward prediction errors (RPEs), a key concept of reinforcement learning theory, are crucial to the formation of episodic memories. In this study, we reveal the neural underpinnings of this process. Using fMRI, we show that signed RPEs (SRPEs) are encoded in the ventral striatum (VS), and crucially, that SRPE VS activity is responsible for the subsequent recollection accuracy of one-shot learned episodic memory associations.

Interplay between midbrain and dorsal anterior cingulate regions arbitrates lingering reward effects on memory encoding

Rewarding events enhance memory encoding via dopaminergic influences on hippocampal plasticity. Phasic dopamine release depends on immediate reward magnitude, but presumably also on tonic dopamine levels, which may vary as a function of the average accumulation of reward over time. Using model-based fMRI in combination with a novel associative memory task, we show that immediate reward magnitude exerts a monotonically increasing influence on the nucleus accumbens, ventral tegmental area (VTA), and hippocampal activity during encoding, and enhances memory. By contrast, average reward levels modulate feedback-related responses in the VTA and hippocampus in a non-linear (inverted U-shape) fashion, with similar effects on memory performance. Additionally, the dorsal anterior cingulate cortex (dACC) monotonically tracks average reward levels, while VTA-dACC functional connectivity is non-linearly modulated (inverted U-shape) by average reward. We propose that the dACC computes the net behavioral impact of average reward and relays this information to memory circuitry via the VTA.

How Curiosity Enhances Hippocampus-Dependent Memory: The Prediction, Appraisal, Curiosity, and Exploration (PACE) Framework

Curiosity plays a fundamental role for learning and memory, but the neural mechanisms that stimulate curiosity and its effect on memory are poorly understood. Accumulating evidence suggests that curiosity states are related to modulations in activity in the dopaminergic circuit and that these modulations impact memory encoding and consolidation for both targets of curiosity and incidental information encountered during curiosity states. To account for this evidence, we propose the Prediction, Appraisal, Curiosity, and Exploration (PACE) framework, which attempts to explain curiosity and memory in terms of cognitive processes, neural circuits, behavior, and subjective experience. The PACE framework generates testable predictions that can stimulate future investigation of the mechanisms underlying curiosity-related memory enhancements.

Reward supports flexible orienting of attention to category information and influences subsequent memory

Preparatory control of attention facilitates the efficient processing and encoding of an expected stimulus. However, this can occur at the expense of increasing the processing cost of unexpected stimuli. Preparatory control can be influenced by motivational factors, such as the expectation of a reward. Interestingly, expectation of a high reward can increase target processing, as well as reduce the cost associated with reorienting. Using a semantic cueing paradigm, we examined the interaction of reward expectation and cue-validity on semantic judgment performance and subsequent memory. Preparatory attention was assessed with pupillometry. Valid category cueing was associated with better semantic judgment performance and better subsequent memory compared to invalidly cued items. Higher reward also resulted in a larger pre-target pupil diameter, which could be indicative of increased preparatory task engagement or arousal. Critically, higher reward also reduced reorienting cost in both semantic judgment and subsequent memory performance. Our findings suggest that reward expectation can facilitate the effective control of preparatory attention for semantic information, and can support optimal goal-directed behavior based on changing task demands.

"A" for Effort: Rewarding Effortful Retrieval Attempts Improves Learning From General Knowledge Errors in Women

Previous research has shown that the prospect of attaining a reward can promote task-engagement, up-regulate attention toward reward-relevant information, and facilitate enhanced encoding of new information into declarative memory. However, past research on reward-based enhancement of declarative memory has focused primarily on paradigms in which rewards are contingent upon accurate responses. Yet, findings from test-enhanced learning show that making errors can also be useful for learning if those errors represent effortful retrieval attempts and are followed by corrective feedback. Here, we used a challenging general knowledge task to examine the effects of explicitly rewarding retrieval effort, defined as a semantically plausible answer to a question (referenced to a semantic knowledge database www.mangelslab.org/bknorms), regardless of response accuracy. In particular, we asked whether intermittent rewards following effortful incorrect responses facilitated learning from corrective feedback as measured by incidental learning outcomes on a 24-48 h delayed retest. Given that effort-contingent extrinsic rewards represent the intersection between an internal locus of control and competency, we compared participants in this "Effort" group to three other groups in a between-subjects design: a Luck group that framed rewards as related to participant-chosen lottery numbers (reward with internal control, not competence-based), a random Award group that framed rewards as computer generated (no control, not competence-based), and a Control group with no reward, but matched on all other task features. Both men and women in the Effort group showed increased self-reports of concentration and positive feelings following the receipt of rewards, as well as subjective effort on the retest, compared to the Control group. However, only women additionally exhibited performance benefits of effort framing on error correction. These benefits were found for both rewarded and non-rewarded trials, but only for correction of low confidence errors, suggesting that effort-contingent rewards produced task-level changes in motivation to learn less familiar information in women, rather than trial-level influences in encoding or consolidation. The Luck and Award groups did not demonstrate significant motivational or behavioral benefits for either gender. These results suggest that both reward context and gender are important factors contributing to the effectiveness of rewards as tools to enhance learning from errors.