Testing the reward prediction error hypothesis with an axiomatic model

Beyond peaks and troughs: Multiplexed performance monitoring signals in the EEG

With the discovery of event-related potentials elicited by errors more than 30 years ago, a new avenue of research on performance monitoring, cognitive control, and decision making emerged. Since then, the field has developed and expanded fulminantly. After a brief overview on the EEG correlates of performance monitoring, this article reviews recent advancements based on single-trial analyses using independent component analysis, multiple regression, and multivariate pattern classification. Given the close interconnection between performance monitoring and reinforcement learning, computational modeling and model-based EEG analyses have made a particularly strong impact. The reviewed findings demonstrate that error- and feedback-related EEG dynamics represent variables reflecting how performance-monitoring signals are weighted and transformed into an adaptation signal that guides future decisions and actions. The model-based single-trial analysis approach goes far beyond conventional peak-and-trough analyses of event-related potentials and enables testing mechanistic theories of performance monitoring, cognitive control, and decision making.

Appetitive conditioning with pornographic stimuli elicits stronger activation in reward regions than monetary and gaming-related stimuli

Appetitive conditioning plays an important role in the development and maintenance of pornography-use and gaming disorders. It is assumed that primary and secondary reinforcers are involved in these processes. Despite the common use of pornography and gaming in the general population appetitive conditioning processes in this context are still not well studied. This study aims to compare appetitive conditioning processes using primary (pornographic) and secondary (monetary and gaming-related) rewards as unconditioned stimuli (UCS) in the general population. Additionally, it investigates the conditioning processes with gaming-related stimuli as this type of UCS was not used in previous studies. Thirty-one subjects participated in a differential conditioning procedure in which four geometric symbols were paired with either pornographic, monetary, or gaming-related rewards or with nothing to become conditioned stimuli (CS + porn, CS + game, CS + money, and CS-) in an functional magnetic resonance imaging study. We observed elevated arousal and valence ratings as well as skin conductance responses for each CS+ condition compared to the CS-. On the neural level, we found activations during the presentation of the CS + porn in the bilateral nucleus accumbens, right medial orbitofrontal cortex, and the right ventral anterior cingulate cortex compared to the CS-, but no significant activations during CS + money and CS + game compared to the CS-. These results indicate that different processes emerge depending on whether primary and secondary rewards are presented separately or together in the same experimental paradigm. Additionally, monetary and gaming-related stimuli seem to have a lower appetitive value than pornographic rewards.

A systems identification approach using Bayes factors to deconstruct the brain bases of emotion regulation

Cognitive reappraisal is fundamental to cognitive therapies and everyday emotion regulation. Analyses using Bayes factors and an axiomatic systems identification approach identified four reappraisal-related components encompassing distributed neural activity patterns across two independent functional magnetic resonance imaging (fMRI) studies (n = 182 and n = 176): (1) an anterior prefrontal system selectively involved in cognitive reappraisal; (2) a fronto-parietal-insular system engaged by both reappraisal and emotion generation, demonstrating a general role in appraisal; (3) a largely subcortical system activated during negative emotion generation but unaffected by reappraisal, including amygdala, hypothalamus and periaqueductal gray; and (4) a posterior cortical system of negative emotion-related regions downregulated by reappraisal. These systems covaried with individual differences in reappraisal success and were differentially related to neurotransmitter binding maps, implicating cannabinoid and serotonin systems in reappraisal. These findings challenge 'limbic'-centric models of reappraisal and provide new systems-level targets for assessing and enhancing emotion regulation.

Additive Effects of Monetary Loss and Positive Emotion in the Human Brain

In many real-life scenarios, our decisions could lead to multiple outcomes that conflict with value. Hence, an appropriate neural representation of the net experienced value of conflicting outcomes, which play a crucial role in guiding future decisions, is critical for adaptive behavior. As some recent functional neuroimaging work has primarily focused on the concurrent processing of monetary gains and aversive information, very little is known regarding the integration of conflicting value signals involving monetary losses and appetitive information in the human brain. To address this critical gap, we conducted a functional MRI study involving healthy human male participants to examine the nature of integrating positive emotion and monetary losses. We employed a novel experimental design where the valence (positive or neutral) of an emotional stimulus indicated the type of outcome (loss or no loss) in a choice task. Specifically, we probed two plausible integration patterns while processing conflicting value signals involving positive emotion and monetary losses: interactive versus additive. We found overlapping main effects of positive (vs neutral) emotion and loss (vs no loss) in multiple brain regions, including the ventromedial prefrontal cortex, striatum, and amygdala, notably with a lack of evidence for interaction. Thus, our findings revealed the additive integration pattern of monetary loss and positive emotion outcomes, suggesting that the experienced value of the monetary loss was not modulated by the valence of the image signaling those outcomes. These findings contribute to our limited understanding of the nature of integrating conflicting outcomes in the healthy human brain with potential clinical relevance.

The parieto-occipital cortex is a candidate neural substrate for the human ability to approximate Bayesian inference

Adaptive decision-making often requires one to infer unobservable states based on incomplete information. Bayesian logic prescribes that individuals should do so by estimating the posterior probability by integrating the prior probability with new information, but the neural basis of this integration is incompletely understood. We record fMRI during a task in which participants infer the posterior probability of a hidden state while we independently modulate the prior probability and likelihood of evidence regarding the state; the task incentivizes participants to make accurate inferences and dissociates expected value from posterior probability. Here we show that activation in a region of left parieto-occipital cortex independently tracks the subjective posterior probability, combining its subcomponents of prior probability and evidence likelihood, and reflecting the individual participants' systematic deviations from objective probabilities. The parieto-occipital cortex is thus a candidate neural substrate for humans' ability to approximate Bayesian inference by integrating prior beliefs with new information.

Dopamine-independent effect of rewards on choices through hidden-state inference

Dopamine is implicated in adaptive behavior through reward prediction error (RPE) signals that update value estimates. There is also accumulating evidence that animals in structured environments can use inference processes to facilitate behavioral flexibility. However, it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine activity, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model where cortex learned to track hidden task states by predicting observations, while basal ganglia learned values and actions via RPEs. This shows that the influence of rewards on choices can stem from dopamine-independent information they convey about the world's state, not the dopaminergic RPEs they produce.

Transdiagnostic inflexible learning dynamics explain deficits in depression and schizophrenia

Deficits in reward learning are core symptoms across many mental disorders. Recent work suggests that such learning impairments arise by a diminished ability to use reward history to guide behaviour, but the neuro-computational mechanisms through which these impairments emerge remain unclear. Moreover, limited work has taken a transdiagnostic approach to investigate whether the psychological and neural mechanisms that give rise to learning deficits are shared across forms of psychopathology. To provide insight into this issue, we explored probabilistic reward learning in patients diagnosed with major depressive disorder (n = 33) or schizophrenia (n = 24) and 33 matched healthy controls by combining computational modelling and single-trial EEG regression. In our task, participants had to integrate the reward history of a stimulus to decide whether it is worthwhile to gamble on it. Adaptive learning in this task is achieved through dynamic learning rates that are maximal on the first encounters with a given stimulus and decay with increasing stimulus repetitions. Hence, over the course of learning, choice preferences would ideally stabilize and be less susceptible to misleading information. We show evidence of reduced learning dynamics, whereby both patient groups demonstrated hypersensitive learning (i.e. less decaying learning rates), rendering their choices more susceptible to misleading feedback. Moreover, there was a schizophrenia-specific approach bias and a depression-specific heightened sensitivity to disconfirmational feedback (factual losses and counterfactual wins). The inflexible learning in both patient groups was accompanied by altered neural processing, including no tracking of expected values in either patient group. Taken together, our results thus provide evidence that reduced trial-by-trial learning dynamics reflect a convergent deficit across depression and schizophrenia. Moreover, we identified disorder distinct learning deficits.

Distinct spatiotemporal brainstem pathways of outcome valence during reward- and punishment-based learning

Learning to seek rewards and avoid punishments, based on positive and negative choice outcomes, is essential for human survival. Yet, the neural underpinnings of outcome valence in the human brainstem and the extent to which they differ in reward and punishment learning contexts remain largely elusive. Here, using simultaneously acquired electroencephalography and functional magnetic resonance imaging data, we show that during reward learning the substantia nigra (SN)/ventral tegmental area (VTA) and locus coeruleus are initially activated following negative outcomes, while the VTA subsequently re-engages exhibiting greater responses for positive than negative outcomes, consistent with an early arousal/avoidance response and a later value-updating process, respectively. During punishment learning, we show that distinct raphe nucleus and SN subregions are activated only by negative outcomes with a sustained post-outcome activity across time, supporting the involvement of these brainstem subregions in avoidance behavior. Finally, we demonstrate that the coupling of these brainstem structures with other subcortical and cortical areas helps to shape participants' serial choice behavior in each context.

Reduced neural encoding of utility prediction errors in cocaine addiction

Influential accounts of addiction posit alterations in adaptive behavior driven by deficient dopaminergic prediction errors (PEs), signaling the discrepancy between actual and expected reward. Dopamine neurons encode these error signals in subjective terms, calibrated by individual risk preferences, as "utility" PEs. It remains unclear, however, whether people with drug addiction have PE deficits or their computational source. Here, using an analogous task to prior single-unit studies with known expectancies, we show that fMRI-measured PEs similarly reflect utility PEs. Relative to control participants, people with chronic cocaine addiction demonstrate reduced utility PEs in the dopaminoceptive ventral striatum, with similar trends in orbitofrontal cortex. Dissecting this PE signal into its subcomponent terms attributed these reductions to weaker striatal responses to received reward/utility, whereas suppression of activity with reward expectation was unchanged. These findings support that addiction may fundamentally disrupt PE signaling and reveal an underappreciated role for perceived reward value in this mechanism.

Negative emotion reduces the discriminability of reward outcomes in the ventromedial prefrontal cortex

Reward and emotion are tightly intertwined, so there is a growing interest in mapping their interactions. However, our knowledge of these interactions in the human brain, especially during the consummatory phase of reward is limited. To address this critical gap, we conducted a functional magnetic resonance imaging study to investigate the effects of negative emotion on reward outcome processing. We employed a novel design where emotional valence (negative or neutral) indicated the type of outcome (reward or no-reward) in a choice task. We focused our functional magnetic resonance imaging analysis on the ventro-medial prefrontal cortex (vmPFC), ventral striatum and amygdala, which were frequently implicated in reward outcome processing. In these regions of interest, we performed multi-voxel pattern analysis to specifically probe how negative emotion modulates reward outcome processing. In vmPFC, using decoding analysis, we found evidence consistent with the reduced discriminability of multi-variate activity patterns of reward vs no-reward outcomes when signaled by a negative relative to a neutral image, suggesting an emotional modulation of reward processing along the plausible common value/valence dimension. These findings advance our limited understanding of the basic brain mechanisms underlying the influence of negative emotion on consummatory reward processing, with potential implications for mental disorders, particularly anxiety and depression.

Auditory and reward structures reflect the pleasure of musical expectancies during naturalistic listening

Enjoying music consistently engages key structures of the neural auditory and reward systems such as the right superior temporal gyrus (R STG) and ventral striatum (VS). Expectations seem to play a central role in this effect, as preferences reliably vary according to listeners' uncertainty about the musical future and surprise about the musical past. Accordingly, VS activity reflects the pleasure of musical surprise, and exhibits stronger correlations with R STG activity as pleasure grows. Yet the reward value of musical surprise - and thus the reason for these surprises engaging the reward system - remains an open question. Recent models of predictive neural processing and learning suggest that forming, testing, and updating hypotheses about one's environment may be intrinsically rewarding, and that the constantly evolving structure of musical patterns could provide ample opportunity for this procedure. Consistent with these accounts, our group previously found that listeners tend to prefer melodic excerpts taken from real music when it either validates their uncertain melodic predictions (i.e., is high in uncertainty and low in surprise) or when it challenges their highly confident ones (i.e., is low in uncertainty and high in surprise). An independent research group (Cheung et al., 2019) replicated these results with musical chord sequences, and identified their fMRI correlates in the STG, amygdala, and hippocampus but not the VS, raising new questions about the neural mechanisms of musical pleasure that the present study seeks to address. Here, we assessed concurrent liking ratings and hemodynamic fMRI signals as 24 participants listened to 50 naturalistic, real-world musical excerpts that varied across wide spectra of computationally modeled uncertainty and surprise. As in previous studies, liking ratings exhibited an interaction between uncertainty and surprise, with the strongest preferences for high uncertainty/low surprise and low uncertainty/high surprise. FMRI results also replicated previous findings, with music liking effects in the R STG and VS. Furthermore, we identify interactions between uncertainty and surprise on the one hand, and liking and surprise on the other, in VS activity. Altogether, these results provide important support for the hypothesized role of the VS in deriving pleasure from learning about musical structure.

The Placebo Effect in Psychosis: Why It Matters and How to Measure It

Psychosis is characterized by unusual percepts and beliefs in the form of hallucinations and delusions. Antipsychotic medication, the primary treatment for psychosis, is often ineffective and accompanied by severe side effects, but research has not identified an effective alternative in several decades. One reason that clinical trials fail is that patients with psychosis tend to show a significant therapeutic response to inert control treatments, known as the placebo effect, which makes it difficult to distinguish drug effects from placebo effects. Conversely, in clinical practice, a strong placebo effect may be useful because it could enhance the overall treatment response. Identifying factors that predict large placebo effects could improve the future outlook of psychosis treatment. Biomarkers of the placebo effect have already been suggested in pain and depression, but not in psychosis. Quantifying markers of the placebo effect would have the potential to predict placebo effects in psychosis clinical trials. Furthermore, the placebo effect and psychosis may represent a shared neurocognitive mechanism in which prior beliefs are weighted against new sensory information to make inferences about reality. Examining this overlap could reveal new insights into the mechanisms underlying psychosis and indicate novel treatment targets. We provide a narrative review of the importance of the placebo effect in psychosis and propose a novel method to assess it.

Economic value in the Brain: A meta-analysis of willingness-to-pay using the Becker-DeGroot-Marschak auction

Forming and comparing subjective values (SVs) of choice options is a critical stage of decision-making. Previous studies have highlighted a complex network of brain regions involved in this process by utilising a diverse range of tasks and stimuli, varying in economic, hedonic and sensory qualities. However, the heterogeneity of tasks and sensory modalities may systematically confound the set of regions mediating the SVs of goods. To identify and delineate the core brain valuation system involved in processing SV, we utilised the Becker-DeGroot-Marschak (BDM) auction, an incentivised demand-revealing mechanism which quantifies SV through the economic metric of willingness-to-pay (WTP). A coordinate-based activation likelihood estimation meta-analysis analysed twenty-four fMRI studies employing a BDM task (731 participants; 190 foci). Using an additional contrast analysis, we also investigated whether this encoding of SV would be invariant to the concurrency of auction task and fMRI recordings. A fail-safe number analysis was conducted to explore potential publication bias. WTP positively correlated with fMRI-BOLD activations in the left ventromedial prefrontal cortex with a sub-cluster extending into anterior cingulate cortex, bilateral ventral striatum, right dorsolateral prefrontal cortex, right inferior frontal gyrus, and right anterior insula. Contrast analysis identified preferential engagement of the mentalizing-related structures in response to concurrent scanning. Together, our findings offer succinct empirical support for the core structures participating in the formation of SV, separate from the hedonic aspects of reward and evaluated in terms of WTP using BDM, and show the selective involvement of inhibition-related brain structures during active valuation.

Opportunity cost determines free-operant action initiation latency and predicts apathy

BACKGROUND

Apathy, a disabling and poorly understood neuropsychiatric symptom, is characterised by impaired self-initiated behaviour. It has been hypothesised that the opportunity cost of time (OCT) may be a key computational variable linking self-initiated behaviour with motivational status. OCT represents the amount of reward which is foregone per second if no action is taken. Using a novel behavioural task and computational modelling, we investigated the relationship between OCT, self-initiation and apathy. We predicted that higher OCT would engender shorter action latencies, and that individuals with greater sensitivity to OCT would have higher behavioural apathy.

METHODS

We modulated the OCT in a novel task called the 'Fisherman Game', Participants freely chose when to self-initiate actions to either collect rewards, or on occasion, to complete non-rewarding actions. We measured the relationship between action latencies, OCT and apathy for each participant across two independent non-clinical studies, one under laboratory conditions (n = 21) and one online (n = 90). 'Average-reward' reinforcement learning was used to model our data. We replicated our findings across both studies.

RESULTS

We show that the latency of self-initiation is driven by changes in the OCT. Furthermore, we demonstrate, for the first time, that participants with higher apathy showed greater sensitivity to changes in OCT in younger adults. Our model shows that apathetic individuals experienced greatest change in subjective OCT during our task as a consequence of being more sensitive to rewards.

CONCLUSIONS

Our results suggest that OCT is an important variable for determining free-operant action initiation and understanding apathy.

A multivariate brain signature for reward

The processing of reinforcers and punishers is crucial to adapt to an ever changing environment and its dysregulation is prevalent in mental health and substance use disorders. While many human brain measures related to reward have been based on activity in individual brain regions, recent studies indicate that many affective and motivational processes are encoded in distributed systems that span multiple regions. Consequently, decoding these processes using individual regions yields small effect sizes and limited reliability, whereas predictive models based on distributed patterns yield larger effect sizes and excellent reliability. To create such a predictive model for the processes of rewards and losses, termed the Brain Reward Signature (BRS), we trained a model to predict the signed magnitude of monetary rewards on the Monetary Incentive Delay task (MID; N = 39) and achieved a highly significant decoding performance (92% for decoding rewards versus losses). We subsequently demonstrate the generalizability of our signature on another version of the MID in a different sample (92% decoding accuracy; N = 12) and on a gambling task from a large sample (73% decoding accuracy, N = 1084). We further provided preliminary data to characterize the specificity of the signature by illustrating that the signature map generates estimates that significantly differ between rewarding and negative feedback (92% decoding accuracy) but do not differ for conditions that differ in disgust rather than reward in a novel Disgust-Delay Task (N = 39). Finally, we show that passively viewing positive and negatively valenced facial expressions loads positively on our signature, in line with previous studies on morbid curiosity. We thus created a BRS that can accurately predict brain responses to rewards and losses in active decision making tasks, and that possibly relates to information seeking in passive observational tasks.

Individuals with problem gambling and obsessive-compulsive disorder learn through distinct reinforcement mechanisms

Obsessive-compulsive disorder (OCD) and pathological gambling (PG) are accompanied by deficits in behavioural flexibility. In reinforcement learning, this inflexibility can reflect asymmetric learning from outcomes above and below expectations. In alternative frameworks, it reflects perseveration independent of learning. Here, we examine evidence for asymmetric reward-learning in OCD and PG by leveraging model-based functional magnetic resonance imaging (fMRI). Compared with healthy controls (HC), OCD patients exhibited a lower learning rate for worse-than-expected outcomes, which was associated with the attenuated encoding of negative reward prediction errors in the dorsomedial prefrontal cortex and the dorsal striatum. PG patients showed higher and lower learning rates for better- and worse-than-expected outcomes, respectively, accompanied by higher encoding of positive reward prediction errors in the anterior insula than HC. Perseveration did not differ considerably between the patient groups and HC. These findings elucidate the neural computations of reward-learning that are altered in OCD and PG, providing a potential account of behavioural inflexibility in those mental disorders.

Neurocomputational mechanisms of affected beliefs

The feedback people receive on their behavior shapes the process of belief formation and self-efficacy in mastering a particular task. However, the neural and computational mechanisms of how the subjective value of self-efficacy beliefs, and the corresponding affect, influence the learning process remain unclear. We investigated these mechanisms during self-efficacy belief formation using fMRI, pupillometry, and computational modeling, and by analyzing individual differences in affective experience. Biases in the formation of self-efficacy beliefs were associated with affect, pupil dilation, and neural activity within the anterior insula, amygdala, ventral tegmental area/ substantia nigra, and mPFC. Specifically, neural and pupil responses mapped the valence of the prediction errors in correspondence with individuals’ experienced affective states and learning biases during self-efficacy belief formation. Together with the functional connectivity dynamics of the anterior insula within this network, our results provide evidence for neural and computational mechanisms of how we arrive at affected beliefs.

Average reward rates enable motivational transfer across independent reinforcement learning tasks

Outcomes and feedbacks on performance may influence behavior beyond the context in which it was received, yet it remains unclear what neurobehavioral mechanisms may account for such lingering influences on behavior. The average reward rate (ARR) has been suggested to regulate motivated behavior, and was found to interact with dopamine-sensitive cognitive processes, such as vigilance and associative memory encoding. The ARR could therefore provide a bridge between independent tasks when these are performed in temporal proximity, such that the reward rate obtained in one task could influence performance in a second subsequent task. Reinforcement learning depends on the coding of prediction error signals by dopamine neurons and their downstream targets, in particular the nucleus accumbens. Because these brain regions also respond to changes in ARR, reinforcement learning may be vulnerable to changes in ARR. To test this hypothesis, we designed a novel paradigm in which participants (n = 245) performed two probabilistic reinforcement learning tasks presented in interleaved trials. The ARR was controlled by an "induction" task which provided feedback with a low (p = 0.58), a medium (p = 0.75), or a high probability of reward (p = 0.92), while the impact of ARR on reinforcement learning was tested by a second "reference" task with a constant reward probability (p = 0.75). We find that performance was significantly lower in the reference task when the induction task provided low reward probabilities (i.e., during low levels of ARR), as compared to the medium and high ARR conditions. Behavioral modeling further revealed that the influence of ARR is best described by models which accumulates average rewards (rather than average prediction errors), and where the ARR directly modulates the prediction error signal (rather than affecting learning rates or exploration). Our results demonstrate how affective information in one domain may transfer and affect motivated behavior in other domains. These findings are particularly relevant for understanding mood disorders, but may also inform abnormal behaviors attributed to dopamine dysfunction.

Asymmetric representation of aversive prediction errors in Pavlovian threat conditioning

Survival in biological environments requires learning associations between predictive sensory cues and threatening outcomes. Such aversive learning may be implemented through reinforcement learning algorithms that are driven by the signed difference between expected and encountered outcomes, termed prediction errors (PEs). While PE-based learning is well established for reward learning, the role of putative PE signals in aversive learning is less clear. Here, we used functional magnetic resonance imaging in humans (21 healthy men and women) to investigate the neural representation of PEs during maintenance of learned aversive associations. Four visual cues, each with a different probability (0, 33, 66, 100%) of being followed by an aversive outcome (electric shock), were repeatedly presented to participants. We found that neural activity at omission (US-) but not occurrence of the aversive outcome (US+) encoded PEs in the medial prefrontal cortex. More expected omission of aversive outcome was associated with lower neural activity. No neural signals fulfilled axiomatic criteria, which specify necessary and sufficient components of PE signals, for signed PE representation in a whole-brain search or in a-priori regions of interest. Our results might suggest that, different from reward learning, aversive learning does not involve signed PE signals that are represented within the same brain region for all conditions.

Individuals with ventromedial frontal damage display unstable but transitive preferences during decision making

The ventromedial frontal lobe (VMF) is important for decision-making, but the precise causal role of the VMF in the decision process has not been fully established. Previous studies have suggested that individuals with VMF damage violate transitivity, a hallmark axiom of rational decisions. However, these prior studies cannot properly distinguish whether individuals with VMF damage are truly prone to choosing irrationally from whether their preferences are simply more variable. We had individuals with focal VMF damage, individuals with other frontal damage, and healthy controls make repeated choices across three categories-artworks, chocolate bar brands, and gambles. Using proper tests of transitivity, we find that, in our study, individuals with VMF damage make rational decisions consistent with transitive preferences, even though they exhibit greater variability in their preferences. That is, the VMF is necessary for having strong and reliable preferences, but not for being a rational decision maker. VMF damage affects the variability with which value is assessed, but not the consistency with which value is sought.

Different brain systems support learning from received and avoided pain during human pain-avoidance learning

Both unexpected pain and unexpected pain absence can drive avoidance learning, but whether they do so via shared or separate neural and neurochemical systems is largely unknown. To address this issue, we combined an instrumental pain-avoidance learning task with computational modeling, functional magnetic resonance imaging (fMRI), and pharmacological manipulations of the dopaminergic (100 mg levodopa) and opioidergic (50 mg naltrexone) systems (N = 83). Computational modeling provided evidence that untreated participants learned more from received than avoided pain. Our dopamine and opioid manipulations negated this learning asymmetry by selectively increasing learning rates for avoided pain. Furthermore, our fMRI analyses revealed that pain prediction errors were encoded in subcortical and limbic brain regions, whereas no-pain prediction errors were encoded in frontal and parietal cortical regions. However, we found no effects of our pharmacological manipulations on the neural encoding of prediction errors. Together, our results suggest that human pain-avoidance learning is supported by separate threat- and safety-learning systems, and that dopamine and endogenous opioids specifically regulate learning from successfully avoided pain.

A unidirectional but not uniform striatal landscape of dopamine signaling for motivational stimuli

Dopamine signals in the striatum are critical for motivated behavior. However, their regional specificity and precise information content are actively debated. Dopaminergic projections to the striatum are topographically organized. Thus, we quantified dopamine release in response to motivational stimuli and associated predictive cues in six principal striatal regions of unrestrained, behaving rats. Absolute signal size and its modulation by stimulus value and by subjective state of the animal were interregionally heterogeneous on a medial to lateral gradient. In contrast, dopamine-concentration direction of change was homogeneous across all regions: appetitive stimuli increased and aversive stimuli decreased dopamine concentration. Although cues predictive of such motivational stimuli acquired the same influence over dopamine homogeneously across all regions, dopamine-mediated prediction-error signals were restricted to the ventromedial, limbic striatum. Together, our findings demonstrate a nuanced striatal landscape of unidirectional but not uniform dopamine signals, topographically encoding distinct aspects of motivational stimuli and their prediction.

Decision neuroscience and neuroeconomics: Recent progress and ongoing challenges

In the past decade, decision neuroscience and neuroeconomics have developed many new insights in the study of decision making. This review provides an overarching update on how the field has advanced in this time period. Although our initial review a decade ago outlined several theoretical, conceptual, methodological, empirical, and practical challenges, there has only been limited progress in resolving these challenges. We summarize significant trends in decision neuroscience through the lens of the challenges outlined for the field and review examples where the field has had significant, direct, and applicable impacts across economics and psychology. First, we review progress on topics including reward learning, explore-exploit decisions, risk and ambiguity, intertemporal choice, and valuation. Next, we assess the impacts of emotion, social rewards, and social context on decision making. Then, we follow up with how individual differences impact choices and new exciting developments in the prediction and neuroforecasting of future decisions. Finally, we consider how trends in decision-neuroscience research reflect progress toward resolving past challenges, discuss new and exciting applications of recent research, and identify new challenges for the field. This article is categorized under: Psychology > Reasoning and Decision Making Psychology > Emotion and Motivation.

Role of anterior midcingulate cortex in self-reward representation and reward allocation judgments within social context

Evaluating rewards for the self and others is essential for social interactions. Previous research has probed the neural substrates signaling rewards in social decision-making tasks as well as the differentiation between self- and other-reward representations. However, studies with different designs have yielded mixed results. After analyzing and comparing previous designs, we differentiated three components in this study: task (reward representation vs. social judgment of reward allocation), agency (self vs. other), and social context (without vs. within). Participants were asked to imagine various share sizes as a proposer in a dictator game during fMRI, and then rated their willingness and preference for these offers in a post-scan behavioral task. To differentiate the regions involved in processing rewards without and within context, we presented the reward to each agent in two sequential frames. Parametric analyses showed that, in the second frame (i.e., within social context), the anterior midcingulate cortex (aMCC) signaled self-reward and preferences for the offer, whereas the right insula tracked the likelihood of proposing the offer. Belief in a just world is positively associated with aMCC responses to self-reward. These results shed light on the role of the aMCC in coding self-reward within the social context to guide social behaviors.

The human insula processes both modality-independent and pain-selective learning signals

Prediction errors (PEs) are generated when there are differences between an expected and an actual event or sensory input. The insula is a key brain region involved in pain processing, and studies have shown that the insula encodes the magnitude of an unexpected outcome (unsigned PEs). In addition to signaling this general magnitude information, PEs can give specific information on the direction of this deviation-i.e., whether an event is better or worse than expected. It is unclear whether the unsigned PE responses in the insula are selective for pain or reflective of a more general processing of aversive events irrespective of modality. It is also unknown whether the insula can process signed PEs at all. Understanding these specific mechanisms has implications for understanding how pain is processed in the brain in both health and in chronic pain conditions. In this study, 47 participants learned associations between 2 conditioned stimuli (CS) with 4 unconditioned stimuli (US; painful heat or loud sound, of one low and one high intensity each) while undergoing functional magnetic resonance imaging (fMRI) and skin conductance response (SCR) measurements. We demonstrate that activation in the anterior insula correlated with unsigned intensity PEs, irrespective of modality, indicating an unspecific aversive surprise signal. Conversely, signed intensity PE signals were modality specific, with signed PEs following pain but not sound located in the dorsal posterior insula, an area implicated in pain intensity processing. Previous studies have identified abnormal insula function and abnormal learning as potential causes of pain chronification. Our findings link these results and suggest that a misrepresentation of learning relevant PEs in the insular cortex may serve as an underlying factor in chronic pain.

Acute stress blunts prediction error signals in the dorsal striatum during reinforcement learning

Acute stress is pervasive in everyday modern life and is thought to affect how people make choices and learn from them. Reinforcement learning, which implicates learning from the unexpected rewarding and punishing outcomes of our choices (i.e., prediction errors), is critical for adjusted behaviour and seems to be affected by acute stress. However, the neural mechanisms by which acute stress disrupts this type of learning are still poorly understood. Here, we investigate whether and how acute stress blunts neural signalling of prediction errors during reinforcement learning using model-based functional magnetic resonance imaging. Male participants completed a well-established reinforcement-learning task involving monetary gains and losses whilst under stress and control conditions. Acute stress impaired participants’ (n = 23) behavioural performance towards obtaining monetary gains (p < 0.001), but not towards avoiding losses (p = 0.57). Importantly, acute stress blunted signalling of prediction errors during gain and loss trials in the dorsal striatum (p = 0.040) — with subsidiary analyses suggesting that acute stress preferentially blunted signalling of positive prediction errors. Our results thus reveal a neurocomputational mechanism by which acute stress may impair reward learning.

Automatic Sensory Predictions: A Review of Predictive Mechanisms in the Brain and Their Link to Conscious Processing

The human brain has the astonishing capacity of integrating streams of sensory information from the environment and forming predictions about future events in an automatic way. Despite being initially developed for visual processing, the bulk of predictive coding research has subsequently focused on auditory processing, with the famous mismatch negativity signal as possibly the most studied signature of a surprise or prediction error (PE) signal. Auditory PEs are present during various consciousness states. Intriguingly, their presence and characteristics have been linked with residual levels of consciousness and return of awareness. In this review we first give an overview of the neural substrates of predictive processes in the auditory modality and their relation to consciousness. Then, we focus on different states of consciousness - wakefulness, sleep, anesthesia, coma, meditation, and hypnosis - and on what mysteries predictive processing has been able to disclose about brain functioning in such states. We review studies investigating how the neural signatures of auditory predictions are modulated by states of reduced or lacking consciousness. As a future outlook, we propose the combination of electrophysiological and computational techniques that will allow investigation of which facets of sensory predictive processes are maintained when consciousness fades away.

Meta-analytic clustering dissociates brain activity and behavior profiles across reward processing paradigms

Reward learning is a ubiquitous cognitive mechanism guiding adaptive choices and behaviors, and when impaired, can lead to considerable mental health consequences. Reward-related functional neuroimaging studies have begun to implicate networks of brain regions essential for processing various peripheral influences (e.g., risk, subjective preference, delay, social context) involved in the multifaceted reward processing construct. To provide a more complete neurocognitive perspective on reward processing that synthesizes findings across the literature while also appreciating these peripheral influences, we used emerging meta-analytic techniques to elucidate brain regions, and in turn networks, consistently engaged in distinct aspects of reward processing. Using a data-driven, meta-analytic, k-means clustering approach, we dissociated seven meta-analytic groupings (MAGs) of neuroimaging results (i.e., brain activity maps) from 749 experimental contrasts across 176 reward processing studies involving 13,358 healthy participants. We then performed an exploratory functional decoding approach to gain insight into the putative functions associated with each MAG. We identified a seven-MAG clustering solution that represented dissociable patterns of convergent brain activity across reward processing tasks. Additionally, our functional decoding analyses revealed that each of these MAGs mapped onto discrete behavior profiles that suggested specialized roles in predicting value (MAG-1 & MAG-2) and processing a variety of emotional (MAG-3), external (MAG-4 & MAG-5), and internal (MAG-6 & MAG-7) influences across reward processing paradigms. These findings support and extend aspects of well-accepted reward learning theories and highlight large-scale brain network activity associated with distinct aspects of reward processing.

Anatomical dissociation of intracerebral signals for reward and punishment prediction errors in humans

Whether maximizing rewards and minimizing punishments rely on distinct brain systems remains debated, given inconsistent results coming from human neuroimaging and animal electrophysiology studies. Bridging the gap across techniques, we recorded intracerebral activity from twenty participants while they performed an instrumental learning task. We found that both reward and punishment prediction errors (PE), estimated from computational modeling of choice behavior, correlate positively with broadband gamma activity (BGA) in several brain regions. In all cases, BGA scaled positively with the outcome (reward or punishment versus nothing) and negatively with the expectation (predictability of reward or punishment). However, reward PE were better signaled in some regions (such as the ventromedial prefrontal and lateral orbitofrontal cortex), and punishment PE in other regions (such as the anterior insula and dorsolateral prefrontal cortex). These regions might therefore belong to brain systems that differentially contribute to the repetition of rewarded choices and the avoidance of punished choices.

Whether maximizing rewards and minimizing punishments rely on distinct brain learning systems remains debated. Here, using intracerebral recordings in humans, the authors provide evidence for brain regions differentially engaged in signaling reward and punishment prediction errors that prescribe repetition versus avoidance of past choices.

The Neural Correlates of Cued Reward Omission

Compared to our understanding of positive prediction error signals occurring due to unexpected reward outcomes, less is known about the neural circuitry in humans that drives negative prediction errors during omission of expected rewards. While classical learning theories such as Rescorla-Wagner or temporal difference learning suggest that both types of prediction errors result from a simple subtraction, there has been recent evidence suggesting that different brain regions provide input to dopamine neurons which contributes to specific components of this prediction error computation. Here, we focus on the brain regions responding to negative prediction error signals, which has been well-established in animal studies to involve a distinct pathway through the lateral habenula. We examine the activity of this pathway in humans, using a conditioned inhibition paradigm with high-resolution functional MRI. First, participants learned to associate a sensory stimulus with reward delivery. Then, reward delivery was omitted whenever this stimulus was presented simultaneously with a different sensory stimulus, the conditioned inhibitor (CI). Both reward presentation and the reward-predictive cue activated midbrain dopamine regions, insula and orbitofrontal cortex. While we found significant activity at an uncorrected threshold for the CI in the habenula, consistent with our predictions, it did not survive correction for multiple comparisons and awaits further replication. Additionally, the pallidum and putamen regions of the basal ganglia showed modulations of activity for the inhibitor that did not survive the corrected threshold.

Reward Learning Through the Lens of RDoC: a Review of Theory, Assessment, and Empirical Findings in the Eating Disorders

PURPOSE OF REVIEW

Reward-related processes may represent important transdiagnostic factors underlying eating pathology. Using the NIMH Research Domain Criteria as a guide, the current article reviews theories, behavioral and self-report assessments, and empirical findings related to reward learning in the eating disorders.

RECENT FINDINGS

Data from behavioral tasks suggest deficits in reinforcement learning, which may become more pronounced with increasing disorder severity and duration. Self-report data strongly implicate positive eating and thinness/restriction expectancies (an element of reward prediction error) in the onset and maintenance of eating pathology. Finally, self-report measures of habit strength demonstrate relationships with eating pathology and illness duration; however, behavioral task data do not support relationships between eating pathology and a propensity towards general habit development. Existing studies are limited, but provide preliminary support for the presence of abnormal reward learning in eating disorders. Continued research is needed to address identified gaps in the literature.

Movement errors during skilled motor performance engage distinct prediction error mechanisms

The brain detects deviations from intended behaviors by estimating the mismatch between predicted and actual outcomes. Axiomatic to these computations are salience and valence prediction error signals, which alert the brain to the occurrence and value of unexpected events. Despite the theoretical assertion of these prediction error signals, it is unknown whether and how brain mechanisms underlying their computations support error processing during skilled motor behavior. Here we demonstrate, with functional magnetic resonance imaging, that internal detection, i.e., without externally-provided feedback, of self-generated movement errors evokes instantaneous activity increases within the salience network and delayed lingering decreases within the nucleus accumbens - a key structure in the reward valuation pathway. A widespread suppression within the sensorimotor network was also observed. Our findings suggest that neural computations of salience and valence prediction errors during skilled motor behaviors operate on different time-scales and, therefore, may contribute differentially to immediate and longer-term adaptive processes.

Behavioral, Physiological, and Neural Signatures of Surprise during Naturalistic Sports Viewing

Surprise signals a discrepancy between past and current beliefs. It is theorized to be linked to affective experiences, the creation of particularly resilient memories, and segmentation of the flow of experience into discrete perceived events. However, the ability to precisely measure naturalistic surprise has remained elusive. We used advanced basketball analytics to derive a quantitative measure of surprise and characterized its behavioral, physiological, and neural correlates in human subjects observing basketball games. We found that surprise was associated with segmentation of ongoing experiences, as reflected by subjectively perceived event boundaries and shifts in neocortical patterns underlying belief states. Interestingly, these effects differed by whether surprising moments contradicted or bolstered current predominant beliefs. Surprise also positively correlated with pupil dilation, activation in subcortical regions associated with dopamine, game enjoyment, and long-term memory. These investigations support key predictions from event segmentation theory and extend theoretical conceptualizations of surprise to real-world contexts.

A Systematic Review of the Positive Valence System in Autism Spectrum Disorder

This review synthesized current literature of behavioral and cognitive studies targeting reward processing in autism spectrum disorder (ASD). The National Institute of Mental Health's Research Domain Criteria (RDoC) Positive Valence System (PVS) domain was used as an overarching framework. The objectives were to determine which component operations of reward processing may be atypical in ASD and consequently postulate a heuristic model of reward processing in ASD that could be evaluated with future research. 34 studies were identified from the Embase, PubMed, PsycINFO, and Web of Science databases and included in the review using guidelines from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (also known as PRISMA guidelines). The extant literature suggested potential relationships between social symptoms of ASD and PVS sub-constructs of reward anticipation, probabilistic and reinforcement learning, reward prediction error, reward (probability), delay, and effort as well as between restricted and repetitive behaviors and interests (RRBIs) and PVS-sub constructs of initial response to reward, reward anticipation, reward (probability), delay, and effort. However, these findings are limited by a sparse and mixed literature for some sub-constructs. We put forward a developmentally informed heuristic model that posits how these component reward processes may be implicated in early ASD behaviors as well as later emerging and more intransigent symptoms. Future research is needed to comprehensively evaluate the proposed model.

Multiple systems in macaques for tracking prediction errors and other types of surprise

Animals learn from the past to make predictions. These predictions are adjusted after prediction errors, i.e., after surprising events. Generally, most reward prediction errors models learn the average expected amount of reward. However, here we demonstrate the existence of distinct mechanisms for detecting other types of surprising events. Six macaques learned to respond to visual stimuli to receive varying amounts of juice rewards. Most trials ended with the delivery of either 1 or 3 juice drops so that animals learned to expect 2 juice drops on average even though instances of precisely 2 drops were rare. To encourage learning, we also included sessions during which the ratio between 1 and 3 drops changed. Additionally, in all sessions, the stimulus sometimes appeared in an unexpected location. Thus, 3 types of surprising events could occur: reward amount surprise (i.e., a scalar reward prediction error), rare reward surprise, and visuospatial surprise. Importantly, we can dissociate scalar reward prediction errors—rewards that deviated from the average reward amount expected—and rare reward events—rewards that accorded with the average reward expectation but that rarely occurred. We linked each type of surprise to a distinct pattern of neural activity using functional magnetic resonance imaging. Activity in the vicinity of the dopaminergic midbrain only reflected surprise about the amount of reward. Lateral prefrontal cortex had a more general role in detecting surprising events. Posterior lateral orbitofrontal cortex specifically detected rare reward events regardless of whether they followed average reward amount expectations, but only in learnable reward environments.

Animals learn from the past to make predictions, and predictions are adjusted whenever surprising outcomes violate those predictions. This study shows that macaques use multiple systems to detect different types of rare reward events; while activity in the vicinity of the dopaminergic midbrain reflected scalar reward prediction errors (i.e. reward amount), activity in the orbitofrontal cortex reflected a rare reward signal.

Distinct signals in medial and lateral VTA dopamine neurons modulate fear extinction at different times

Dopamine (DA) neurons are thought to encode reward prediction error (RPE), in addition to other signals, such as salience. While RPE is known to support learning, the role of salience in learning remains less clear. To address this, we recorded and manipulated VTA DA neurons in mice during fear extinction. We applied deep learning to classify mouse freezing behavior, eliminating the need for human scoring. Our fiber photometry recordings showed DA neurons in medial and lateral VTA have distinct activity profiles during fear extinction: medial VTA activity more closely reflected RPE, while lateral VTA activity more closely reflected a salience-like signal. Optogenetic inhibition of DA neurons in either region slowed fear extinction, with the relevant time period for inhibition differing across regions. Our results indicate salience-like signals can have similar downstream consequences to RPE-like signals, although with different temporal dependencies.

How mood tunes prediction: a neurophenomenological account of mood and its disturbance in major depression

In this article, we propose a neurophenomenological account of what moods are, and how they work. We draw upon phenomenology to show how mood attunes a person to a space of significant possibilities. Mood structures a person's lived experience by fixing the kinds of significance the world can have for them in a given situation. We employ Karl Friston's free-energy principle to show how this phenomenological concept of mood can be smoothly integrated with cognitive neuroscience. We will argue that mood is a consequence of acting in the world with the aim of minimizing expected free energy-a measure of uncertainty about the future consequences of actions. Moods summarize how the organism is faring overall in its predictive engagements, tuning the organism's expectations about how it is likely to fare in the future. Agents that act to minimize expected free energy will have a feeling of how well or badly they are doing at maintaining grip on the multiple possibilities that matter to them. They will have what we will call a 'feeling of grip' that structures the possibilities they are ready to engage with over long time-scales, just as moods do.

Amphetamine disrupts haemodynamic correlates of prediction errors in nucleus accumbens and orbitofrontal cortex

In an uncertain world, the ability to predict and update the relationships between environmental cues and outcomes is a fundamental element of adaptive behaviour. This type of learning is typically thought to depend on prediction error, the difference between expected and experienced events and in the reward domain that has been closely linked to mesolimbic dopamine. There is also increasing behavioural and neuroimaging evidence that disruption to this process may be a cross-diagnostic feature of several neuropsychiatric and neurological disorders in which dopamine is dysregulated. However, the precise relationship between haemodynamic measures, dopamine and reward-guided learning remains unclear. To help address this issue, we used a translational technique, oxygen amperometry, to record haemodynamic signals in the nucleus accumbens (NAc) and orbitofrontal cortex (OFC), while freely moving rats performed a probabilistic Pavlovian learning task. Using a model-based analysis approach to account for individual variations in learning, we found that the oxygen signal in the NAc correlated with a reward prediction error, whereas in the OFC it correlated with an unsigned prediction error or salience signal. Furthermore, an acute dose of amphetamine, creating a hyperdopaminergic state, disrupted rats' ability to discriminate between cues associated with either a high or a low probability of reward and concomitantly corrupted prediction error signalling. These results demonstrate parallel but distinct prediction error signals in NAc and OFC during learning, both of which are affected by psychostimulant administration. Furthermore, they establish the viability of tracking and manipulating haemodynamic signatures of reward-guided learning observed in human fMRI studies by using a proxy signal for BOLD in a freely behaving rodent.

Signals of anticipation of reward and of mean reward rates in the human brain

Theoretical models of dopamine function stemming from reinforcement learning theory have emphasized the importance of prediction errors, which signal changes in the expectation of impending rewards. Much less is known about the effects of mean reward rates, which may be of motivational significance due to their role in computing the optimal effort put into exploiting reward opportunities. Here, we used a reinforcement learning model to design three functional neuroimaging studies and disentangle the effects of changes in reward expectations and mean reward rates, showing recruitment of specific regions in the brainstem regardless of prediction errors. While changes in reward expectations activated ventral striatal areas as in previous studies, mean reward rates preferentially modulated the substantia nigra/ventral tegmental area, deep layers of the superior colliculi, and a posterior pontomesencephalic region. These brainstem structures may work together to set motivation and attentional efforts levels according to perceived reward opportunities.

Breaking human social decision making into multiple components and then putting them together again

Most of our waking time as human beings is spent interacting with other individuals. In order to make good decisions in this social milieu, it is often necessary to make inferences about the internal states, traits and intentions of others. Recently, some progress has been made toward uncovering the neural computations underlying human social decision-making by combining functional magnetic resonance neuroimaging (fMRI) with computational modeling of behavior. Modeling of behavioral data allows us to identify the key computations necessary for social decision-making and to determine how these computations are integrated. Furthermore, by correlating these variables against neuroimaging data, it has become possible to elucidate where in the brain various computations are implemented. Here we review the current state of knowledge in the domain of social computational neuroscience. Findings to date have emphasized that social decisions are driven by multiple computations conducted in parallel, and implemented in distinct brain regions. We suggest that further progress is going to depend on identifying how and where such variables get integrated in order to yield a coherent behavioral output.

Decreased transfer of value to action in Tourette syndrome

OBJECTIVE

Tourette syndrome is a neurodevelopmental disorder putatively associated with a hyperdopaminergic state. Therefore, it seems plausible that excessive dopamine transmission in Tourette syndrome alters the ability to learn based on rewards and punishments. We tested whether Tourette syndrome patients exhibited altered reinforcement learning and corresponding feedback-related EEG deflections.

METHODS

We used a reinforcement learning task providing factual and counterfactual feedback in a sample of 15 Tourette syndrome patients and matched healthy controls whilst recording EEG. The paradigm presented various reward probabilities to enforce adaptive adjustments. We employed a computational model to derive estimates of the prediction error, which we used for single-trial regression analysis of the EEG data.

RESULTS

We found that Tourette syndrome patients showed increased choice stochasticity compared to controls. The feedback-related negativity represented an axiomatic prediction error for factual feedback and did not differ between groups. We observed attenuated P3a modulation specifically for factual feedback in Tourette syndrome patients, representing impaired coding of attention allocation.

CONCLUSION

Our findings indicate that cortical prediction error coding is unaffected by Tourette syndrome. Nonetheless, the transfer of learned values into choice formation is degraded, in line with a hyperdopaminergic state.

Beyond Reward Prediction Errors: Human Striatum Updates Rule Values During Learning

Humans naturally group the world into coherent categories defined by membership rules. Rules can be learned implicitly by building stimulus-response associations using reinforcement learning or by using explicit reasoning. We tested if the striatum, in which activation reliably scales with reward prediction error, would track prediction errors in a task that required explicit rule generation. Using functional magnetic resonance imaging during a categorization task, we show that striatal responses to feedback scale with a "surprise" signal derived from a Bayesian rule-learning model and are inconsistent with RL prediction error. We also find that striatum and caudal inferior frontal sulcus (cIFS) are involved in updating the likelihood of discriminative rules. We conclude that the striatum, in cooperation with the cIFS, is involved in updating the values assigned to categorization rules when people learn using explicit reasoning.

Multiple associative structures created by reinforcement and incidental statistical learning mechanisms

Learning the structure of the world can be driven by reinforcement but also occurs incidentally through experience. Reinforcement learning theory has provided insight into how prediction errors drive updates in beliefs but less attention has been paid to the knowledge resulting from such learning. Here we contrast associative structures formed through reinforcement and experience of task statistics. BOLD neuroimaging in human volunteers demonstrates rigid representations of rewarded sequences in temporal pole and posterior orbito-frontal cortex, which are constructed backwards from reward. By contrast, medial prefrontal cortex and a hippocampal-amygdala border region carry reward-related knowledge but also flexible statistical knowledge of the currently relevant task model. Intriguingly, ventral striatum encodes prediction error responses but not the full RL- or statistically derived task knowledge. In summary, representations of task knowledge are derived via multiple learning processes operating at different time scales that are associated with partially overlapping and partially specialized anatomical regions.

Associative learning occurs through reinforcement mechanisms as well as incidentally through experience of statistical relationships. Here, the authors report that these two learning processes are associated with specialized anatomical regions that operate at different time scales.

Distinct Processing of Aversive Experience in Amygdala Subregions

Background

The amygdala is an anatomically complex medial temporal brain structure whose subregions are considered to serve distinct functions. However, their precise role in mediating human aversive experience remains ill understood.

Methods

We used functional magnetic resonance imaging in 39 healthy volunteers with varying levels of trait anxiety to assess distinct contributions of the basolateral amygdala (BLA) and centromedial amygdala to anticipation and experience of aversive events. Additionally, we examined the relationship between any identified functional subspecialization and measures of subjective reported aversion and trait anxiety.

Results

Our results show that the centromedial amygdala is responsive to aversive outcomes but insensitive to predictive aversive cues. In contrast, the BLA encodes an aversive prediction error that quantifies whether cues and outcomes are worse than expected. A neural representation within the BLA for distinct threat levels was mirrored in self-reported subjective anxiety across individuals. Furthermore, high trait-anxious individuals were characterized by indiscriminately heightened BLA activity in response to aversive cues, regardless of actual threat level.

Conclusions

Our results demonstrate that amygdala subregions are distinctly engaged in processing of aversive experience, with elevated and undifferentiated BLA responses to threat emerging as a potential neurobiological mediator of vulnerability to anxiety disorders.

Short-term mindfulness practice attenuates reward prediction errors signals in the brain

Activity changes in dopaminergic neurons encode the ongoing discrepancy between expected and actual value of a stimulus, providing a teaching signal for a reward prediction process. Previous work comparing a cohort of long-term Zen meditators to controls demonstrated an attenuation of reward prediction signals to appetitive reward in the striatum. Using a cross-commodity design encompassing primary- and secondary-reward conditioning experiments, the present study asks the question of whether reward prediction signals are causally altered by mindfulness training in naïve subjects. Volunteers were randomly assigned to 8 weeks of mindfulness training (MT), active control training (CT), or a one-time mindfulness induction group (MI). We observed a decreased response to positive prediction errors in the putamen in the MT group compared to CT using both a primary and a secondary-reward experiment. Furthermore, the posterior insula showed greater activation to primary rewards, independently of their predictability, in the MT group, relative to CT and MI group. These results support the notion that increased attention to the present moment and its interoceptive features - a core component of mindfulness practice - may reduce predictability effects in reward processing, without dampening (in fact, enhancing) the response to the actual delivery of the stimulus.

Annual Research Review: Developmental computational psychiatry

Most psychiatric disorders emerge during childhood and adolescence. This is also a period that coincides with the brain undergoing substantial growth and reorganisation. However, it remains unclear how a heightened vulnerability to psychiatric disorder relates to this brain maturation. Here, we propose 'developmental computational psychiatry' as a framework for linking brain maturation to cognitive development. We argue that through modelling some of the brain's fundamental cognitive computations, and relating them to brain development, we can bridge the gap between brain and cognitive development. This in turn can lead to a richer understanding of the ontogeny of psychiatric disorders. We illustrate this perspective with examples from reinforcement learning and dopamine function. Specifically, we show how computational modelling deepens an understanding of how cognitive processes, such as reward learning, effort learning, and social learning might go awry in psychiatric disorders. Finally, we sketch the promises and limitations of a developmental computational psychiatry.

Musical reward prediction errors engage the nucleus accumbens and motivate learning

Enjoying music reliably ranks among life's greatest pleasures. Like many hedonic experiences, it engages several reward-related brain areas, with activity in the nucleus accumbens (NAc) most consistently reflecting the listener's subjective response. Converging evidence suggests that this activity arises from musical "reward prediction errors" (RPEs) that signal the difference between expected and perceived musical events, but this hypothesis has not been directly tested. In the present fMRI experiment, we assessed whether music could elicit formally modeled RPEs in the NAc by applying a well-established decision-making protocol designed and validated for studying RPEs. In the scanner, participants chose between arbitrary cues that probabilistically led to dissonant or consonant music, and learned to make choices associated with the consonance, which they preferred. We modeled regressors of trial-by-trial RPEs, finding that NAc activity tracked musically elicited RPEs, to an extent that explained variance in the individual learning rates. These results demonstrate that music can act as a reward, driving learning and eliciting RPEs in the NAc, a hub of reward- and music enjoyment-related activity.

Habits without values

Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).