Signed reward prediction errors drive declarative learning

Reward-related regions play a role in natural story comprehension

The reward system was shown to be involved in a wide array of processes. Nevertheless, the exploration of the involvement of the reward system during language processing has not yet been directly tested. We investigated the role of reward-processing regions while listening to a natural story. We utilized a published dataset in which half of the participants listened to a natural story and the others listened to a scrambled version of it to compare the functional MRI signals in the reward system between these conditions and discovered a distinct pattern between conditions. This suggests that the reward system is activated during the comprehension of natural stories. We also show evidence that the fMRI signals in reward-related areas might potentially correlate with the predictability level of processed sentences. Further research is needed to determine the nature of the involvement and the way the activity interacts with various aspects of the sentences.

Reward conditioning may not have an effect on category-specific memory

Behavioural tagging facilitates the temporary storage of seemingly insignificant episodic events, which may later become salient and enhanced in memory. Human behavioural studies have demonstrated selective memory enhancement for neutral stimuli from one category when this category is subsequently paired with reward. Although this phenomenon has implications for the role of reward conditioning on emotional and adaptive memory, its generalisability is underexplored. We conducted four experiments to investigate whether pairing items from a semantic category, animals or objects, with high or low rewards resulted in preferential memory for the high-reward category. Three of these experiments also aimed to replicate the category-specific retrospective enhancement effect reported by Patil et al. and two explored the corresponding prospective memory effect. None of our experiments showed consistent evidence for an effect of reward on category-specific memory enhancement, despite employing the same reward paradigm and incidental encoding protocol as in the original study. Consequently, we found no evidence for category-specific retrospective or prospective enhancement effects. Our experiments were conducted online which is an equally relevant method for assessing behavioural phenomenon as the in-person studies conducted by Patil et al. Overall, our results question the generalisability of previously reported category-specific memory enhancement effects due to reward.

Deliberately making miskates: Behavioural consistency under win maximization and loss maximization conditions

We argue that the feedback traditionally used to indicate negative outcomes causes future detrimental performance because of the default goal of win maximization. In gaming paradigms where participants intentionally performed as well (win maximization) and as poorly (loss maximization) as possible, we showed a double dissociation where actions following wins were more consistent during win maximization, but actions following losses were more consistent during loss maximization. This broader distinction between goal-congruent and goal-incongruent feedback suggests that individuals are able to flexibly redefine their definition of 'success', and provide a reconsideration of the way we think about 'losing'.

Long-term, multi-event surprise correlates with enhanced autobiographical memory

Neurobiological and psychological models of learning emphasize the importance of prediction errors (surprises) for memory formation. This relationship has been shown for individual momentary surprising events; however, it is less clear whether surprise that unfolds across multiple events and timescales is also linked with better memory of those events. We asked basketball fans about their most positive and negative autobiographical memories of individual plays, games and seasons, allowing surprise measurements spanning seconds, hours and months. We used advanced analytics on National Basketball Association play-by-play data and betting odds spanning 17 seasons, more than 22,000 games and more than 5.6 million plays to compute and align the estimated surprise value of each memory. We found that surprising events were associated with better recall of positive memories on the scale of seconds and months and negative memories across all three timescales. Game and season memories could not be explained by surprise at shorter timescales, suggesting that long-term, multi-event surprise correlates with memory. These results expand notions of surprise in models of learning and reinforce its relevance in real-world domains.

Remembering unexpected beauty: Contributions of the ventral striatum to the processing of reward prediction errors regarding the facial attractiveness in face memory

The COVID-19 pandemic has led people to predict facial attractiveness from partially covered faces. Differences in the predicted and observed facial attractiveness (i.e., masked and unmasked faces, respectively) are defined as reward prediction error (RPE) in a social context. Cognitive neuroscience studies have elucidated the neural mechanisms underlying RPE-induced memory improvements in terms of monetary rewards. However, little is known about the mechanisms underlying RPE-induced memory modulation in terms of social rewards. To elucidate this, the present functional magnetic resonance imaging (fMRI) study investigated activity and functional connectivity during face encoding. In encoding trials, participants rated the predicted attractiveness of faces covered except for around the eyes (prediction phase) and then rated the observed attractiveness of these faces without any cover (outcome phase). The difference in ratings between these phases was defined as RPE in facial attractiveness, and RPE was categorized into positive RPE (increased RPE from the prediction to outcome phases), negative RPE (decreased RPE from the prediction to outcome phases), and non-RPE (no difference in RPE between the prediction and outcome phases). During retrieval, participants were presented with individual faces that had been seen and unseen in the encoding trials, and were required to judge whether or not each face had been seen in the encoding trials. Univariate activity in the ventral striatum (VS) exhibited a linear increase with increased RPE in facial attractiveness. In the multivariate pattern analysis (MVPA), activity patterns in the VS and surrounding areas (extended VS) significantly discriminated between positive/negative RPE and non-RPE. In the functional connectivity analysis, significant functional connectivity between the extended VS and the hippocampus was observed most frequently in positive RPE. Memory improvements by face-based RPE could be involved in functional networks between the extended VS (representing RPE) and the hippocampus, and the interaction could be modulated by RPE values in a social context.

Signed and unsigned effects of prediction error on memory: Is it a matter of choice?

Adaptive decision-making is governed by at least two types of memory processes. On the one hand, learned predictions through integrating multiple experiences, and on the other hand, one-shot episodic memories. These two processes interact, and predictions - particularly prediction errors - influence how episodic memories are encoded. However, studies using computational models disagree on the exact shape of this relationship, with some findings showing an effect of signed prediction errors and others showing an effect of unsigned prediction errors on episodic memory. We argue that the choice-confirmation bias, which reflects stronger learning from choice-confirming compared to disconfirming outcomes, could explain these seemingly diverging results. Our perspective implies that the influence of prediction errors on episodic encoding critically depends on whether people can freely choose between options (i.e., instrumental learning tasks) or not (Pavlovian learning tasks). The choice-confirmation bias on memory encoding might have evolved to prioritize memory representations that optimize reward-guided decision-making. We conclude by discussing open issues and implications for future studies.

The effect of prediction error on episodic memory encoding is modulated by the outcome of the predictions

Expectations can lead to prediction errors of varying degrees depending on the extent to which the information encountered in the environment conforms with prior knowledge. While there is strong evidence on the computationally specific effects of such prediction errors on learning, relatively less evidence is available regarding their effects on episodic memory. Here, we had participants work on a task in which they learned context/object-category associations of different strengths based on the outcomes of their predictions. We then used a reinforcement learning model to derive subject-specific trial-to-trial estimates of prediction error at encoding and link it to subsequent recognition memory. Results showed that model-derived prediction errors at encoding influenced subsequent memory as a function of the outcome of participants' predictions (correct vs. incorrect). When participants correctly predicted the object category, stronger prediction errors (as a consequence of weak expectations) led to enhanced memory. In contrast, when participants incorrectly predicted the object category, stronger prediction errors (as a consequence of strong expectations) led to impaired memory. These results highlight the important moderating role of choice outcome that may be related to interactions between the hippocampal and striatal dopaminergic systems.

How a token-based game may elicit the reward prediction error and increase engagement of students in elementary school. A pilot study

Student engagement is essential to academic success and student-wellbeing. In the past, fostering engagement though extrinsic rewards has often been found to be of limited effectiveness over the long term. However, extrinsic rewards are important for improving engagement with non-intrinsically rewarding activities. Thus, in the present study a mechanism that is meant to prolong the effects of extrinsic rewards was investigated: the reward prediction error. This error occurs when rewards are awarded contrary to the awardee’s expectations. In a quasi-experiment, 39 elementary school students participated in a classroom-based game, which was supposed to motivate them to solve math exercises. It combined reinforcement with elements of luck, which were supposed to elicit the reward prediction error. After 2 weeks, the intervention group had completed significantly more math exercises compared to a pretest and, importantly, also more correctly solved exercises than a control group. This suggests that game-based reinforcement that elicits the reward prediction error might help to increase student engagement over the medium term. It furthermore highlights the importance of applying gamification elements not only digitally but also in analog settings.

Midbrain-Hippocampus Structural Connectivity Selectively Predicts Motivated Memory Encoding

Motivation is a powerful driver of learning and memory. Functional MRI studies show that interactions among the dopaminergic midbrain substantia nigra/ventral tegmental area (SN/VTA), hippocampus, and nucleus accumbens (NAc) are critical for motivated memory encoding. However, it is not known whether these effects are transient and purely functional, or whether individual differences in the structure of this circuit underlie motivated memory encoding. To quantify individual differences in structure, diffusion-weighted MRI and probabilistic tractography were used to quantify SN/VTA-striatum and SN/VTA-hippocampus pathways associated with motivated memory encoding in humans. Male and female participants completed a motivated source memory paradigm. During encoding, words were randomly assigned to one of three conditions, reward ($1.00), control ($0.00), or punishment (-$1.00). During retrieval, participants were asked to retrieve item and source information of the previously studied words and were rewarded or penalized according to their performance. Source memory for words assigned to both reward and punishment conditions was greater than those for control words, but there were no differences in item memory based on value. Anatomically, probabilistic tractography results revealed a heterogeneous, topological arrangement of the SN/VTA. Tract density measures of SN/VTA-hippocampus pathways were positively correlated with individual differences in reward-and-punishment-modulated memory performance, whereas density of SN/VTA-striatum pathways showed no association. This novel finding suggests that pathways emerging from the human SV/VTA are anatomically separable and functionally heterogeneous. Individual differences in structural connectivity of the dopaminergic hippocampus-VTA loop are selectively associated with motivated memory encoding.SIGNIFICANCE STATEMENT Functional MRI studies show that interactions among the SN/VTA, hippocampus, and NAc are critical for motivated memory encoding. This has led to competing theories that posit either SN/VTA-NAc reward prediction errors or SN/VTA-hippocampus signals underlie motivated memory encoding. Additionally, it is not known whether these effects are transient and purely functional or whether individual differences in the structure of these circuits underlie motivated memory encoding. Using diffusion-weighted MRI and probabilistic tractography, we show that tract density measures of SN/VTA-hippocampus pathways are positively correlated with motivated memory performance, whereas density of SN/VTA-striatum pathways show no association. This finding suggests that anatomic individual differences of the dopaminergic hippocampus-VTA loop are selectively associated with motivated memory encoding.

Assessing behavioural profiles following neutral, positive and negative feedback

Previous data suggest zero-value, neutral outcomes (draw) are subjectively assigned negative rather than positive valence. The combined observations of faster rather than slower reaction times, subsequent actions defined by shift rather than stay behaviour, reduced flexibility, and, larger rather than smaller deviations from optimal performance following draws all align with the consequences of explicitly negative outcomes such as losses. We further tested the relationships between neutral, positive and negative outcomes by manipulating value salience and observing their behavioural profiles. Despite speeded reaction times and a non-significant bias towards shift behaviour similar to losses when draws were assigned the value of 0 (Experiment 1), the degree of shift behaviour approached an approximation of optimal performance when the draw value was explicitly positive (+1). This was in contrast to when the draw value was explicitly negative (-1), which led to a significant increase in the degree of shift behaviour (Experiment 2). Similar modifications were absent when the same value manipulations were applied to win or lose trials (Experiment 3). Rather than viewing draws as neutral and valence-free outcomes, the processing cascade generated by draws produces a complex behavioural profile containing elements found in response to both explicitly positive and explicitly negative results.

Mnemonic prediction errors promote detailed memories

When our experience violates our predictions, it is adaptive to update our knowledge to promote a more accurate representation of the world and facilitate future predictions. Theoretical models propose that these mnemonic prediction errors should be encoded into a distinct memory trace to prevent interference with previous, conflicting memories. We investigated this proposal by repeatedly exposing participants to pairs of sequentially presented objects (A → B), thus evoking expectations. Then, we violated participants' expectations by replacing the second object in the pairs with a novel object (A → C). The following item memory test required participants to discriminate between identical old items and similar lures, thus testing detailed and distinctive item memory representations. In two experiments, mnemonic prediction errors enhanced item memory: Participants correctly identified more old items as old when those items violated expectations during learning, compared with items that did not violate expectations. This memory enhancement for C items was only observed when participants later showed intact memory for the related A → B pairs, suggesting that strong predictions are required to facilitate memory for violations. Following up on this, a third experiment reduced prediction strength prior to violation and subsequently eliminated the memory advantage of violations. Interestingly, mnemonic prediction errors did not increase gist-based mistakes of identifying old items as similar lures or identifying similar lures as old. Enhanced item memory in the absence of gist-based mistakes suggests that violations enhanced memory for items' details, which could be mediated via distinct memory traces. Together, these results advance our knowledge of how mnemonic prediction errors promote memory formation.

The benefits of impossible tests: Assessing the role of error-correction in the pretesting effect

Relative to studying alone, guessing the meanings of unknown words can improve later recognition of their meanings, even if those guesses were incorrect – the pretesting effect (PTE). The error-correction hypothesis suggests that incorrect guesses produce error signals that promote memory for the meanings when they are revealed. The current research sought to test the error-correction explanation of the PTE. In three experiments, participants studied unfamiliar Finnish-English word pairs by either studying each complete pair or by guessing the English translation before its presentation. In the latter case, the participants also guessed which of two categories the word belonged to. Hence, guesses from the correct category were semantically closer to the true translation than guesses from the incorrect category. In Experiment 1, guessing increased subsequent recognition of the English translations, especially for translations that were presented on trials in which the participants’ guesses were from the correct category. Experiment 2 replicated these target recognition effects while also demonstrating that they do not extend to associative recognition performance. Experiment 3 again replicated the target recognition pattern, while also examining participants’ metacognitive recognition judgments. Participants correctly judged that their memory would be better after small than after large errors, but incorrectly believed that making any errors would be detrimental, relative to study-only. Overall, the data are inconsistent with the error-correction hypothesis; small, within-category errors produced better recognition than large, cross-category errors. Alternative theories, based on elaborative encoding and motivated learning, are considered.

Dopamine modulations of reward-driven music memory consolidation

Music listening provides one of the most significant abstract rewards for humans because hearing music activates the dopaminergic mesolimbic system. Given the strong link between reward, dopamine, and memory, we aimed here to investigate the hypothesis that dopamine-dependent musical reward can drive memory improvements. Twenty-nine healthy participants of both sexes provided reward ratings of unfamiliar musical excerpts that had to be remembered following a consolidation period under three separate conditions: after the ingestion of a dopaminergic antagonist, a dopaminergic precursor, or a placebo. Linear mixed modeling of the intervention data showed that the effect of reward on memory-i.e., the greater the reward experienced while listening to the musical excerpts, the better the memory recollection performance-was modulated by both dopaminergic signaling and individual differences in reward processing. Greater pleasure was consistently associated with better memory outcomes in participants with high sensitivity to musical reward, but this effect was lost when dopaminergic signaling was disrupted in participants with average or low musical hedonia. Our work highlights the flexibility of the human dopaminergic system, which can enhance memory formation not only through explicit and/or primary reinforcers but also via abstract and aesthetic rewards such as music.

Reward prediction errors drive declarative learning irrespective of agency

Recent years have witnessed a steady increase in the number of studies investigating the role of reward prediction errors (RPEs) in declarative learning. Specifically, in several experimental paradigms, RPEs drive declarative learning, with larger and more positive RPEs enhancing declarative learning. However, it is unknown whether this RPE must derive from the participant's own response, or whether instead, any RPE is sufficient to obtain the learning effect. To test this, we generated RPEs in the same experimental paradigm where we combined an agency and a nonagency condition. We observed no interaction between RPE and agency, suggesting that any RPE (irrespective of its source) can drive declarative learning. This result holds implications for declarative learning theory.

Contextual prediction errors reorganize naturalistic episodic memories in time

Episodic memories are contextual experiences ordered in time. This is underpinned by associative binding between events within the same contexts. The role of prediction errors in declarative memory is well established but has not been investigated in the time dimension of complex episodic memories. Here we combine these two properties of episodic memory, extend them into the temporal domain and demonstrate that prediction errors in different naturalistic contexts lead to changes in the temporal ordering of event structures in them. The wrongly predicted older sequences were weakened despite their reactivation. Interestingly the newly encoded sequences with prediction errors, seen once, showed accuracy as high as control sequences which were viewed repeatedly without change. Drift-diffusion modelling revealed a lower decision threshold for the newer sequences than older sequences, reflected by their faster recall. Moreover, participants' adjustments to their decision threshold significantly correlated with their relative speed of sequence memory recall. These results suggest a temporally distinct and adaptive role for prediction errors in learning and reorganizing episodic temporal sequences.

Individual differences in experienced and observational decision-making illuminate interactions between reinforcement learning and declarative memory

Decision making can be shaped both by trial-and-error experiences and by memory of unique contextual information. Moreover, these types of information can be acquired either by means of active experience or by observing others behave in similar situations. The interactions between reinforcement learning parameters that inform decision updating and memory formation of declarative information in experienced and observational learning settings are, however, unknown. In the current study, participants took part in a probabilistic decision-making task involving situations that either yielded similar outcomes to those of an observed player or opposed them. By fitting alternative reinforcement learning models to each subject, we discerned participants who learned similarly from experience and observation from those who assigned different weights to learning signals from these two sources. Participants who assigned different weights to their own experience versus those of others displayed enhanced memory performance as well as subjective memory strength for episodes involving significant reward prospects. Conversely, memory performance of participants who did not prioritize their own experience over others did not seem to be influenced by reinforcement learning parameters. These findings demonstrate that interactions between implicit and explicit learning systems depend on the means by which individuals weigh relevant information conveyed via experience and observation.

Signed and unsigned reward prediction errors dynamically enhance learning and memory

Memory helps guide behavior, but which experiences from the past are prioritized? Classic models of learning posit that events associated with unpredictable outcomes as well as, paradoxically, predictable outcomes, deploy more attention and learning for those events. Here, we test reinforcement learning and subsequent memory for those events, and treat signed and unsigned reward prediction errors (RPEs), experienced at the reward-predictive cue or reward outcome, as drivers of these two seemingly contradictory signals. By fitting reinforcement learning models to behavior, we find that both RPEs contribute to learning by modulating a dynamically changing learning rate. We further characterize the effects of these RPE signals on memory and show that both signed and unsigned RPEs enhance memory, in line with midbrain dopamine and locus-coeruleus modulation of hippocampal plasticity, thereby reconciling separate findings in the literature.

A predictive account of how novelty influences declarative memory

A rich body of studies in the human and non-human literature has examined the question how novelty influences memory. For a variety of different stimuli, ranging from simple objects and words to vastly complex scenarios, the literature reports that novelty improves memory in some cases, but impairs memory in other cases. In recent attempts to reconcile these conflicting findings, novelty has been divided into different subtypes, such as relative versus absolute novelty, or stimulus versus contextual novelty. Nevertheless, a single overarching theory of novelty and memory has been difficult to attain, probably due to the complexities in the interactions among stimuli, environmental factors (e.g., spatial and temporal context) and level of prior knowledge (but see Duszkiewicz et al., 2019; Kafkas & Montaldi, 2018b; Schomaker & Meeter, 2015). Here we describe how a predictive coding framework might be able to shed new light on different types of novelty and how they affect declarative memory in humans. More precisely, we consider how prior expectations modulate the influence of novelty on encoding episodes into memory, e.g., in terms of surprise, and how novelty/surprise affect memory for surrounding information. By reviewing a range of behavioural findings and their possible underlying neurobiological mechanisms, we highlight where a predictive coding framework succeeds and where it appears to struggle.

Signed Reward Prediction Errors in the Ventral Striatum Drive Episodic Memory

Recent behavioral evidence implicates reward prediction errors (RPEs) as a key factor in the acquisition of episodic memory. Yet, important neural predictions related to the role of RPEs in episodic memory acquisition remain to be tested. Humans (both sexes) performed a novel variable-choice task where we experimentally manipulated RPEs and found support for key neural predictions with fMRI. Our results show that in line with previous behavioral observations, episodic memory accuracy increases with the magnitude of signed (i.e., better/worse-than-expected) RPEs (SRPEs). Neurally, we observe that SRPEs are encoded in the ventral striatum (VS). Crucially, we demonstrate through mediation analysis that activation in the VS mediates the experimental manipulation of SRPEs on episodic memory accuracy. In particular, SRPE-based responses in the VS (during learning) predict the strength of subsequent episodic memory (during recollection). Furthermore, functional connectivity between task-relevant processing areas (i.e., face-selective areas) and hippocampus and ventral striatum increased as a function of RPE value (during learning), suggesting a central role of these areas in episodic memory formation. Our results consolidate reinforcement learning theory and striatal RPEs as key factors subtending the formation of episodic memory.SIGNIFICANCE STATEMENT Recent behavioral research has shown that reward prediction errors (RPEs), a key concept of reinforcement learning theory, are crucial to the formation of episodic memories. In this study, we reveal the neural underpinnings of this process. Using fMRI, we show that signed RPEs (SRPEs) are encoded in the ventral striatum (VS), and crucially, that SRPE VS activity is responsible for the subsequent recollection accuracy of one-shot learned episodic memory associations.

Failure to modulate reward prediction errors in declarative learning with theta (6 Hz) frequency transcranial alternating current stimulation

Recent evidence suggests that reward prediction errors (RPEs) play an important role in declarative learning, but its neurophysiological mechanism remains unclear. Here, we tested the hypothesis that RPEs modulate declarative learning via theta-frequency oscillations, which have been related to memory encoding in prior work. For that purpose, we examined the interaction between RPE and transcranial Alternating Current Stimulation (tACS) in declarative learning. Using a between-subject (real versus sham stimulation group), single-blind stimulation design, 76 participants learned 60 Dutch-Swahili word pairs, while theta-frequency (6 Hz) tACS was administered over the medial frontal cortex (MFC). Previous studies have implicated MFC in memory encoding. We replicated our previous finding of signed RPEs (SRPEs) boosting declarative learning; with larger and more positive RPEs enhancing memory performance. However, tACS failed to modulate the SRPE effect in declarative learning and did not affect memory performance. Bayesian statistics supported evidence for an absence of effect. Our study confirms a role of RPE in declarative learning, but also calls for standardized procedures in transcranial electrical stimulation.

Reward prediction errors create event boundaries in memory

We remember when things change. Particularly salient are experiences where there is a change in rewards, eliciting reward prediction errors (RPEs). How do RPEs influence our memory of those experiences? One idea is that this signal directly enhances the encoding of memory. Another, not mutually exclusive, idea is that the RPE signals a deeper change in the environment, leading to the mnemonic separation of subsequent experiences from what came before, thereby creating a new latent context and a more separate memory trace. We tested this in four experiments where participants learned to predict rewards associated with a series of trial-unique images. High-magnitude RPEs indicated a change in the underlying distribution of rewards. To test whether these large RPEs created a new latent context, we first assessed recognition priming for sequential pairs that included a high-RPE event or not (Exp. 1: n = 27 & Exp. 2: n = 83). We found evidence of recognition priming for the high-RPE event, indicating that the high-RPE event is bound to its predecessor in memory. Given that high-RPE events are themselves preferentially remembered (Rouhani, Norman, & Niv, 2018), we next tested whether there was an event boundary across a high-RPE event (i.e., excluding the high-RPE event itself; Exp. 3: n = 85). Here, sequential pairs across a high RPE no longer showed recognition priming whereas pairs within the same latent reward state did, providing initial evidence for an RPE-modulated event boundary. We then investigated whether RPE event boundaries disrupt temporal memory by asking participants to order and estimate the distance between two events that had either included a high-RPE event between them or not (Exp. 4). We found (n = 49) and replicated (n = 77) worse sequence memory for events across a high RPE. In line with our recognition priming results, we did not find sequence memory to be impaired between the high-RPE event and its predecessor, but instead found worse sequence memory for pairs across a high-RPE event. Moreover, greater distance between events at encoding led to better sequence memory for events across a low-RPE event, but not a high-RPE event, suggesting separate mechanisms for the temporal ordering of events within versus across a latent reward context. Altogether, these findings demonstrate that high-RPE events are both more strongly encoded, show intact links with their predecessor, and act as event boundaries that interrupt the sequential integration of events. We captured these effects in a variant of the Context Maintenance and Retrieval model (CMR; Polyn, Norman, & Kahana, 2009), modified to incorporate RPEs into the encoding process.

Reward Prediction Error and Declarative Memory

Learning based on reward prediction error (RPE) was originally proposed in the context of nondeclarative memory. We postulate that RPE may support declarative memory as well. Indeed, recent years have witnessed a number of independent empirical studies reporting effects of RPE on declarative memory. We provide a brief overview of these studies, identify emerging patterns, and discuss open issues such as the role of signed versus unsigned RPEs in declarative learning.

Behavioural and neural interactions between objective and subjective performance in a Matching Pennies game

To examine the behavioural and neural interactions between objective and subjective performance during competitive decision-making, participants completed a Matching Pennies game where win-rates were fixed within three conditions (win > lose, win = lose, win < lose) and outcomes were predicted at each trial. Using random behaviour as the hallmark of optimal performance, we observed item (heads), contingency (win-stay, lose-shift) and combinatorial (HH, HT, TH, TT) biases across all conditions. Higher-quality behaviour represented by a reduction in combinatorial bias was observed during high win-rate exposure. In contrast, over-optimism biases were observed only in conditions where win rates were equal to, or less than, loss rates. At a group level, a neural measure of outcome evaluation (feedback-related negativity; FRN) indexed the binary distinction between positive and negative outcome. At an individual level, increased belief in successful performance accentuated FRN amplitude differences between wins and losses. Taken together, the data suggest that objective experiences of, or, subjective beliefs in, the predominance of positive outcomes may be mutual attempts to self-regulate performance during competition. In this way, increased exposure to positive outcomes (real or imagined) may help to weight the output of the more diligent and analytic System 2, relative to the impulsive and intuitive System 1.

Positive reward prediction errors during decision-making strengthen memory encoding

Dopamine is thought to provide reward prediction error signals to temporal lobe memory systems, but the role of these signals in episodic memory has not been fully characterized. Here we developed an incidental memory paradigm to (i) estimate the influence of reward prediction errors on the formation of episodic memories, (ii) dissociate this influence from surprise and uncertainty, (iii) characterize the role of temporal correspondence between prediction error and memoranda presentation and (iv) determine the extent to which this influence is dependent on memory consolidation. We found that people encoded incidental memoranda more strongly when they gambled for potential rewards. Moreover, the degree to which gambling strengthened encoding scaled with the reward prediction error experienced when memoranda were presented (and not before or after). This encoding enhancement was detectable within minutes and did not differ substantially after 24 h, indicating that it is not dependent on memory consolidation. These results suggest a computationally and temporally specific role for reward prediction error signalling in memory formation.

The sound of beauty: How complexity determines aesthetic preference

Stimulus complexity is an important determinant of aesthetic preference. An influential idea is that increases in stimulus complexity lead to increased preference up to an optimal point after which preference decreases (inverted-U pattern). However, whereas some studies indeed observed this pattern, most studies instead showed an increased preference for more complexity. One complicating issue is that it remains unclear how to define complexity. To address this, we approached complexity and its relation to aesthetic preference from a predictive coding perspective. Here, low- and high-complexity stimuli would correspond to low and high levels of prediction errors, respectively. We expected participants to prefer stimuli which are neither too easy to predict (low prediction error), nor too difficult (high prediction error). To test this, we presented two sequences of tones on each trial that varied in predictability from highly regular (low prediction error) to completely random (high prediction error), and participants had to indicate which of the two sequences they preferred in a two-interval forced-choice task. The complexity of each tone sequence (amount of prediction error) was estimated using entropy. Results showed that participants tended to choose stimuli with intermediate complexity over those of high or low complexity. This confirms the century-old idea that stimulus complexity has an inverted-U relationship to aesthetic preference.

Expected Value of Reward Predicts Episodic Memory for Incidentally Learnt Reward-Item Associations

In this paper, we draw connections between reward processing and cognition by behaviourally testing the implications of neurobiological theories of reward processing on memory. Single-cell neurophysiology in non-human primates and imaging work in humans suggests that the dopaminergic reward system responds to different components of reward: expected value; outcome or prediction error; and uncertainty of reward (Schultz et al., 2008). The literature on both incidental and motivated learning has focused on understanding how expected value and outcome—linked to increased activity in the reward system—lead to consolidation-related memory enhancements. In the current study, we additionally investigate the impact of reward uncertainty on human memory. The contribution of reward uncertainty—the spread of the reward probability distribution irrespective of the magnitude—has not been previously examined. To examine the effects of uncertainty on memory, a word-learning task was introduced, along with a surprise delayed recognition memory test. Using Bayesian model selection, we found evidence only for expected value as a predictor of memory performance. Our findings suggest that reward uncertainty does not enhance memory for individual items. This supports emerging evidence that an effect of uncertainty on memory is only observed in high compared to low risk environments.

Oscillatory signatures of reward prediction errors in declarative learning

Reward prediction errors (RPEs) are crucial to learning. Whereas these mismatches between reward expectation and reward outcome are known to drive procedural learning, their role in declarative learning remains underexplored. Earlier work from our lab addressed this, and consistently found that signed reward prediction errors (SRPEs; "better-than-expected" signals) boost declarative learning. In the current EEG study, we sought to explore the neural signatures of SRPEs. Participants studied 60 Dutch-Swahili word pairs while RPE magnitudes were parametrically manipulated. Behaviorally, we replicated our previous findings that SRPEs drive declarative learning, with increased recognition for word pairs accompanied by large, positive RPEs. In the EEG data, at the start of reward feedback processing, we found an oscillatory (theta) signature consistent with unsigned reward prediction errors (URPEs; "different-than-expected" signals). Slightly later during reward feedback processing, we observed oscillatory (high-beta and high-alpha) signatures for SRPEs during reward feedback, similar to SRPE signatures during procedural learning. These findings illuminate the time course of neural oscillations in processing reward during declarative learning, providing important constraints for future theoretical work.