Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Early-Life Adversities Are Associated With Lower Expected Value Signaling in the Adult Brain

BACKGROUND

Early adverse experiences are assumed to affect fundamental processes of reward learning and decision making. However, computational neuroimaging studies investigating these circuits in the context of adversity are sparse and limited to studies conducted in adolescent samples, leaving the long-term effects unexplored.

METHODS

Using data from a longitudinal birth cohort study (n = 156; 87 female), we investigated associations between adversities and computational markers of reward learning (i.e., expected value, prediction errors). At age 33 years, all participants completed a functional magnetic resonance imaging-based passive avoidance task. Psychopathology measures were collected at the time of functional magnetic resonance imaging investigation and during the COVID-19 pandemic. We applied a principal component analysis to capture common variations across 7 adversity measures. The resulting adversity factors (factor 1: postnatal psychosocial adversities and prenatal maternal smoking; factor 2: prenatal maternal stress and obstetric adversity; factor 3: lower maternal stimulation) were linked with psychopathology and neural responses in the core reward network using multiple regression analysis.

RESULTS

We found that the adversity dimension primarily informed by lower maternal stimulation was linked to lower expected value representation in the right putamen, right nucleus accumbens, and anterior cingulate cortex. Expected value encoding in the right nucleus accumbens further mediated the relationship between this adversity dimension and psychopathology and predicted higher withdrawn symptoms during the COVID-19 pandemic.

CONCLUSIONS

Our results suggested that early adverse experiences in caregiver context might have a long-term disruptive effect on reward learning in reward-related brain regions, which can be associated with suboptimal decision making and thereby may increase the vulnerability of developing psychopathology.

Functional Abnormality of the Reward System in Depressed Adolescents and Young Adults with and without Suicidal Behavior

OBJECTIVE

To identify local and functional connectivity abnormalities in the brain's reward network in depressed adolescents and young adults with and without suicidal behavior.

METHODS

Magnetic resonance imaging data were obtained from 41 major depressive disorder (MDD) patients with suicidal behavior (sMDD, males/females: 12/29), 44 MDD patients without suicidal behavior (nMDD, males/females: 13/32), and 52 healthy controls (HCs, males/females: 17/35). The Young Mania Scale, Hamilton Depression Scale, Columbia Suicide Scale, and Scale for Suicide Ideation were used to evaluate emotional state and suicidal ideation and behaviors. The amplitude of low frequency fluctuations (ALFF), regional homogeneity (ReHo) and functional connectivity of 11 regions of interest (ROIs) in the reward network were determined.

RESULTS

ALFF values in the vmPFC of the nMDD group were significantly lower than those in the HC group (p = 0.031). The ReHo values of the nMDD group were lower in the lVS but higher in the vmPFC than those of the HC group (P = 0.018 and 0.025, respectively). Functional connectivity of the AC with the vmPFC, lVS, rVS, and vmPFC was increased in the sMDD group compared with that in the nMDD group (P = 0.038, 0.034, 0.006, respectively).

CONCLUSION

Local and functional connectivity abnormalities in the reward network were found in the MDD groups. However, increased functional connectivity was found in only the sMDD group.

From Complexity to Precision-Charting Decision-Making Through Normative Modeling

This Viewpoint synthesizes data-driven and theory-driven approaches to normative modeling.

Trajectory of reward-related abnormalities in unaffected relatives of patients with bipolar disorder - A longitudinal fMRI study

First-degree relatives of patients with bipolar disorder are at heightened risk of mood episodes, which may be attributed to the existence of endophenotypes i.e., heritable (neuro)biological changes present in patients and their unaffected relatives (UR). In this longitudinal MRI study, we aim to investigate the trajectories of aberrant reward-related functional changes identified in UR vs healthy controls (HC). Sixty-eight UR and 65 HC of similar age and gender distribution underwent MRI at baseline while performing a card guessing task. Of these, 29 UR and 36 HC were investigated with the same protocol following a 16-month period in average. We first identified brain regions showing group differences in the neural response to expected value (EV) and reward prediction error (PE) at baseline and analyzed how the reward-related response in these regions changed over time in UR vs HC. Relative to HC at baseline, UR showed lower EV signal in the right ventrolateral prefrontal cortex (vlPFC) and paracingulate gyrus and lower PE signal in the left vlPFC and dorsomedial PFC. The trajectories of these abnormalities in UR showed a normalization of the prefrontal EV signals, whereas the PE signals which correlated with depressive symptoms remained stable over time. While the UR showed both blunted EV and PE signals, none of these abnormalities increased over time, which is consistent with the observed stable mood symptoms.

The neural correlates of memory integration in value-based decision-making during human spatial navigation

In daily life, we often make decisions based on relative value of the options, and we often derive these values from segmenting or integrating the outcomes of past episodes in memory. The neural correlates involved in value-based decision-making have been extensively studied in the literature, but few studies have investigated this topic in decisions that require segmenting or integrating episodic memory from related sources, and even fewer studies examine it in the context of spatial navigation. Building on the computational models from our previous studies, the current study investigates the neural substrates involved in decisions that require people either segment or integrate wayfinding outcomes involving different goals, across virtual spatial navigation tasks with differing demands. We find that when decisions require computation of spatial distances for navigation options, but also evaluation of one's prior spatial navigation ability with the task, the estimated value of navigational choices (EV) modulates neural activity in the dorsomedial prefrontal (dmPFC) cortex and ventrolateral prefrontal (vlFPC) cortex. However, superior parietal cortex tracked EV when decision-making tasks only require spatial distance memory but not evaluation of spatial navigation ability. Our findings reveal divergent neural substrates of memory integration in value-based decision-making under different spatial processing demands.

Postnatal Phencyclidine-Induced Deficits in Decision Making Are Ameliorated by Optogenetic Inhibition of Ventromedial Orbitofrontal Cortical Glutamate Neurons

Background

The orbitofrontal cortex (OFC) is essential for decision making, and functional disruptions within the OFC are evident in schizophrenia. Postnatal phencyclidine (PCP) administration in rats is a neurodevelopmental manipulation that induces schizophrenia-relevant cognitive impairments. We aimed to determine whether manipulating OFC glutamate cell activity could ameliorate postnatal PCP-induced deficits in decision making.

Methods

Male and female Wistar rats (n = 110) were administered saline or PCP on postnatal days 7, 9, and 11. In adulthood, we expressed YFP (yellow fluorescent protein) (control), ChR2 (channelrhodopsin-2) (activation), or eNpHR 3.0 (enhanced halorhodopsin) (inhibition) in glutamate neurons within the ventromedial OFC (vmOFC). Rats were tested on the probabilistic reversal learning task once daily for 20 days while we manipulated the activity of vmOFC glutamate cells. Behavioral performance was analyzed using a Q-learning computational model of reinforcement learning.

Results

Compared with saline-treated rats expressing YFP, PCP-treated rats expressing YFP completed fewer reversals, made fewer win-stay responses, and had lower learning rates. We induced similar performance impairments in saline-treated rats by activating vmOFC glutamate cells (ChR2). Strikingly, PCP-induced performance deficits were ameliorated when the activity of vmOFC glutamate cells was inhibited (halorhodopsin).

Conclusions

Postnatal PCP-induced deficits in decision making are associated with hyperactivity of vmOFC glutamate cells. Thus, normalizing vmOFC activity may represent a potential therapeutic target for decision-making deficits in patients with schizophrenia.

Neural Correlates of Social Decision-Making

Objective: Recent studies have utilized innovative techniques to investigate the neural mechanisms underlying social and individual decision-making, aiming to understand how individuals respond to the world. Method : In this review, we summarized current scientific evidence concerning the neural underpinnings of social decision-making and their impact on social behavior. Results: Critical brain regions involved in social cognition and decision-making are integral to the process of social decision-making. Notably, the medial prefrontal cortex (mPFC) and temporoparietal junction (TPJ) contribute to the comprehension of others' mental states. Similarly, the posterior superior temporal sulcus (pSTS) shows heightened activity when individuals observe faces and movements. On the lateral surface of the brain, the inferior frontal gyrus (IFG) and inferior parietal sulcus (IPS) play a role in social cognition. Furthermore, the medial surface of the brain, including the amygdala, anterior cingulate cortex (ACC), and anterior insula (AI), also participates in social cognition processes. Regarding decision-making, functional magnetic resonance imaging (fMRI) studies have illuminated the involvement of a network of brain regions, encompassing the ventromedial prefrontal cortex (vmPFC), ventral striatum (VS), and nucleus accumbens (NAcc). Conclusion: Dysfunction in specific subregions of the prefrontal cortex (PFC) has been linked to various psychiatric conditions. These subregions play pivotal roles in cognitive, emotional, and social processing, and their impairment can contribute to the development and manifestation of psychiatric symptoms. A comprehensive understanding of the unique contributions of these PFC subregions to psychiatric disorders has the potential to inform the development of targeted interventions and treatments for affected individuals.

A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI

Introduction

Computational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses.

Methods

Simulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses.

Results

Initial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise.

Conclusion

Overall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder.

The neurocognitive role of working memory load when Pavlovian motivational control affects instrumental learning

Research suggests that a fast, capacity-limited working memory (WM) system and a slow, incremental reinforcement learning (RL) system jointly contribute to instrumental learning. Thus, situations that strain WM resources alter instrumental learning: under WM loads, learning becomes slow and incremental, the reliance on computationally efficient learning increases, and action selection becomes more random. It is also suggested that Pavlovian learning influences people's behavior during instrumental learning by providing hard-wired instinctive responses including approach to reward predictors and avoidance of punishment predictors. However, it remains unknown how constraints on WM resources affect instrumental learning under Pavlovian influence. Thus, we conducted a functional magnetic resonance imaging (fMRI) study (N = 49) in which participants completed an instrumental learning task with Pavlovian-instrumental conflict (the orthogonalized go/no-go task) both with and without extra WM load. Behavioral and computational modeling analyses revealed that WM load reduced the learning rate and increased random choice, without affecting Pavlovian bias. Model-based fMRI analysis revealed that WM load strengthened RPE signaling in the striatum. Moreover, under WM load, the striatum showed weakened connectivity with the ventromedial and dorsolateral prefrontal cortex when computing reward expectations. These results suggest that the limitation of cognitive resources by WM load promotes slow and incremental learning through the weakened cooperation between WM and RL; such limitation also makes action selection more random, but it does not directly affect the balance between instrumental and Pavlovian systems.

The right superior temporal gyrus plays a role in semantic-rule learning: Evidence supporting a reinforcement learning model

In real-life communication, individuals use language that carries evident rewarding and punishing elements, such as praise and criticism. A common trend is to seek more praise while avoiding criticism. Furthermore, semantics is crucial for conveying information, but such semantic access to native and foreign languages is subtly distinct. To investigate how rule learning occurs in different languages and to highlight the importance of semantics in this process, we investigated both verbal and non-verbal rule learning in first (L1) and second (L2) languages using a reinforcement learning framework, including a semantic rule and a color rule. Our computational modeling on behavioral and brain imaging data revealed that individuals may be more motivated to learn and adhere to rules in an L1 compared to L2, with greater striatum activation during the outcome phase in the L1. Additionally, results on the learning rates and inverse temperature in the two rule learning tasks showed that individuals tend to be conservative and are reluctant to change their judgments regarding rule learning of semantic information. Moreover, the greater the prediction errors, the greater activation of the right superior temporal gyrus in the semantic-rule learning condition, demonstrating that such learning has differential neural correlates than symbolic rule learning. Overall, the findings provide insight into the neural mechanisms underlying rule learning in different languages, and indicate that rule learning involving verbal semantics is not a general symbolic learning that resembles a conditioned stimulus-response, but rather has its own specific characteristics.

Auditory and reward structures reflect the pleasure of musical expectancies during naturalistic listening

Enjoying music consistently engages key structures of the neural auditory and reward systems such as the right superior temporal gyrus (R STG) and ventral striatum (VS). Expectations seem to play a central role in this effect, as preferences reliably vary according to listeners' uncertainty about the musical future and surprise about the musical past. Accordingly, VS activity reflects the pleasure of musical surprise, and exhibits stronger correlations with R STG activity as pleasure grows. Yet the reward value of musical surprise - and thus the reason for these surprises engaging the reward system - remains an open question. Recent models of predictive neural processing and learning suggest that forming, testing, and updating hypotheses about one's environment may be intrinsically rewarding, and that the constantly evolving structure of musical patterns could provide ample opportunity for this procedure. Consistent with these accounts, our group previously found that listeners tend to prefer melodic excerpts taken from real music when it either validates their uncertain melodic predictions (i.e., is high in uncertainty and low in surprise) or when it challenges their highly confident ones (i.e., is low in uncertainty and high in surprise). An independent research group (Cheung et al., 2019) replicated these results with musical chord sequences, and identified their fMRI correlates in the STG, amygdala, and hippocampus but not the VS, raising new questions about the neural mechanisms of musical pleasure that the present study seeks to address. Here, we assessed concurrent liking ratings and hemodynamic fMRI signals as 24 participants listened to 50 naturalistic, real-world musical excerpts that varied across wide spectra of computationally modeled uncertainty and surprise. As in previous studies, liking ratings exhibited an interaction between uncertainty and surprise, with the strongest preferences for high uncertainty/low surprise and low uncertainty/high surprise. FMRI results also replicated previous findings, with music liking effects in the R STG and VS. Furthermore, we identify interactions between uncertainty and surprise on the one hand, and liking and surprise on the other, in VS activity. Altogether, these results provide important support for the hypothesized role of the VS in deriving pleasure from learning about musical structure.

Human thalamic low-frequency oscillations correlate with expected value and outcomes during reinforcement learning

Reinforcement-based adaptive decision-making is believed to recruit fronto-striatal circuits. A critical node of the fronto-striatal circuit is the thalamus. However, direct evidence of its involvement in human reinforcement learning is lacking. We address this gap by analyzing intra-thalamic electrophysiological recordings from eight participants while they performed a reinforcement learning task. We found that in both the anterior thalamus (ATN) and dorsomedial thalamus (DMTN), low frequency oscillations (LFO, 4-12 Hz) correlated positively with expected value estimated from computational modeling during reward-based learning (after outcome delivery) or punishment-based learning (during the choice process). Furthermore, LFO recorded from ATN/DMTN were also negatively correlated with outcomes so that both components of reward prediction errors were signaled in the human thalamus. The observed differences in the prediction signals between rewarding and punishing conditions shed light on the neural mechanisms underlying action inhibition in punishment avoidance learning. Our results provide insight into the role of thalamus in reinforcement-based decision-making in humans.

Dopamine regulates decision thresholds in human reinforcement learning in males

Dopamine fundamentally contributes to reinforcement learning, but recent accounts also suggest a contribution to specific action selection mechanisms and the regulation of response vigour. Here, we examine dopaminergic mechanisms underlying human reinforcement learning and action selection via a combined pharmacological neuroimaging approach in male human volunteers (n = 31, within-subjects; Placebo, 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist Haloperidol). We found little credible evidence for previously reported beneficial effects of L-dopa vs. Haloperidol on learning from gains and altered neural prediction error signals, which may be partly due to differences experimental design and/or drug dosages. Reinforcement learning drift diffusion models account for learning-related changes in accuracy and response times, and reveal consistent decision threshold reductions under both drugs, in line with the idea that lower dosages of D2 receptor antagonists increase striatal DA release via an autoreceptor-mediated feedback mechanism. These results are in line with the idea that dopamine regulates decision thresholds during reinforcement learning, and may help to bridge action selection and response vigor accounts of dopamine.

Impulse control disorder in Parkinson's disease is associated with abnormal frontal value signalling

Dopaminergic medication is well established to boost reward- versus punishment-based learning in Parkinson's disease. However, there is tremendous variability in dopaminergic medication effects across different individuals, with some patients exhibiting much greater cognitive sensitivity to medication than others. We aimed to unravel the mechanisms underlying this individual variability in a large heterogeneous sample of early-stage patients with Parkinson's disease as a function of comorbid neuropsychiatric symptomatology, in particular impulse control disorders and depression. One hundred and ninety-nine patients with Parkinson's disease (138 ON medication and 61 OFF medication) and 59 healthy controls were scanned with functional MRI while they performed an established probabilistic instrumental learning task. Reinforcement learning model-based analyses revealed medication group differences in learning from gains versus losses, but only in patients with impulse control disorders. Furthermore, expected-value related brain signalling in the ventromedial prefrontal cortex was increased in patients with impulse control disorders ON medication compared with those OFF medication, while striatal reward prediction error signalling remained unaltered. These data substantiate the hypothesis that dopamine's effects on reinforcement learning in Parkinson's disease vary with individual differences in comorbid impulse control disorder and suggest they reflect deficient computation of value in medial frontal cortex, rather than deficient reward prediction error signalling in striatum. See Michael Browning (https://doi.org/10.1093/brain/awad248) for a scientific commentary on this article.

Subtypes in addiction and their neurobehavioral profiles across three functional domains

Rates of return to use in addiction treatment remain high. We argue that the development of improved treatment options will require advanced understanding of individual heterogeneity in Substance Use Disorders (SUDs). We hypothesized that considerable individual differences exist in the three functional domains underlying addiction-approach-related behavior, executive function, and negative emotionality. We included N = 593 participants from the enhanced Nathan Kline Institute-Rockland Sample community sample (ages 18-59, 67% female) that included N = 420 Controls and N = 173 with past SUDs [54% female; N = 75 Alcohol Use Disorder (AUD) only, N = 30 Cannabis Use Disorder (CUD) only, and N = 68 Multiple SUDs]. To test our a priori hypothesis that distinct neuro-behavioral subtypes exist within individuals with past SUDs, we conducted a latent profile analysis with all available phenotypic data as input (74 subscales from 18 measures), and then characterized resting-state brain function for each discovered subtype. Three subtypes with distinct neurobehavioral profiles were recovered (p < 0.05, Cohen's D: 0.4-2.8): a "Reward type" with higher approach-related behavior (N = 69); a "Cognitive type" with lower executive function (N = 70); and a "Relief type" with high negative emotionality (N = 34). For those in the Reward type, substance use mapped onto resting-state connectivity in the Value/Reward, Ventral-Frontoparietal and Salience networks; for the Cognitive type in the Auditory, Parietal Association, Frontoparietal and Salience networks; and for the Relief type in the Parietal Association, Higher Visual and Salience networks (pFDR < 0.05). Subtypes were equally distributed amongst individuals with different primary SUDs (χ2 = 4.71, p = 0.32) and gender (χ2 = 3.44, p = 0.18). Results support functionally derived subtypes, demonstrating considerable individual heterogeneity in the multi-dimensional impairments in addiction. This confirms the need for mechanism-based subtyping to inform the development of personalized addiction medicine approaches.

Alcohol and cannabinoid binges and daily exposure to nicotine in adolescent/young adult rats induce sex-dependent long-term appetitive instrumental learning impairment

Adolescence is a critical developmental period, concerning anatomical, neurochemical and behavioral changes. Moreover, adolescents are more sensitive to the long-term deleterious effects of drug abuse. Binge-like consumption of alcohol and marijuana, along with tobacco smoking, is a dangerous pattern often observed in adolescents during weekends. Nevertheless, the long-term effect of their adolescent co-exposure has not been yet experimentally investigated. Long-Evans adolescent male (n = 20) and female (n = 20) rats from postnatal day 30 (P30) until P60 were daily treated with nicotine (0.3 mg/kg, i.p.), and, on two consecutive 'binging days' per week (for a total of eight times), received an intragastric ethanol solution (3 g/kg) and an intraperitoneal (i.p.) dose of cannabinoid 1/2 receptor agonist WIN55,212-2 (1.2 mg/kg). These rats were tested after treatment discontinuation at > P90 for associative food-rewarded operant learning in the two-lever conditioning chambers for six consecutive days on a fixed ratio 1 (FR1) schedule followed by another six days of daily FR2 schedule testing, after 42 days rest. We found the main effects of sex x treatment interactions in FR1 but not in FR2 experiments. Treated females show attenuated operant responses for food pellets during all FR1 and the FR2 schedule, whilst the treated males show an impairment in FR2 but not in the FR1 schedule. Moreover, the treated females' percentage of learners was significantly lower than female controls in FR1 while treated males were lower than controls in FR2. Our findings suggest that intermittent adolescent abuse of common drugs, such as alcohol and marijuana, and chronic tobacco exposure can cause significant long-term effects on motivation for natural reinforcers later in adulthood in both sexes. Females appear to be sensitive earlier to the deleterious effects of adolescent polydrug abuse, with both sexes having an increased likelihood of developing lifelong brain alterations.

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders

In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.

Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T

The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.

Abnormal Brain Networks Related to Drug and Nondrug Reward Anticipation and Outcome Processing in Stimulant Use Disorder: A Functional Connectomics Approach

BACKGROUND

Drug addiction is associated with blunted neural responses to nondrug rewards, such as money, but heightened responses to drug cues that predict drug-reward outcomes. This dissociation underscores the role of incentive context in the attribution of motivational salience, which may reflect a narrowing toward drug-related goals. This hypothesis, however, has scarcely been investigated.

METHODS

To address this important scientific gap, the current study performed an empirical assessment of differences in salience attribution by comparing patients with stimulant use disorder (SUD) (n = 41) with control participants (n = 48) on network connectivity related to anticipation and outcome processing using a modified monetary incentive delay task. We hypothesized increased task-related activation and connectivity to drug rewards in patients with SUD, and reduced task-related activation and connectivity to monetary rewards during incentive processing across brain networks.

RESULTS

In the presence of behavioral and regional brain activation similarities, we found that patients with SUD showed significantly less connectivity involving three separate distributed networks during monetary reward anticipation, and drug and monetary reward outcome processing. No group connectivity differences for drug reward anticipation were identified. Additional graph theory analyses revealed that patients with SUD had longer path lengths across these networks, all of which positively correlated with the duration of stimulant drug use.

CONCLUSIONS

Specific disruptions in connectivity in networks related to the anticipation of nondrug reward together with more general dysconnectivity in the processing of rewarding outcomes suggest an insensitivity to consequences. These observations support the notion of a predominance of habitual control in patients with SUD.

Trajectory of aberrant reward processing in patients with bipolar disorder - A longitudinal fMRI study

BACKGROUND

Bipolar disorder (BD), and especially the mania phenotype, is characterized by heightened reward responsivity and aberrant reward processing. In this longitudinal fMRI study, we investigated neuronal response during reward anticipation as the computed expected value (EV) and outcome evaluation as reward prediction error (RPE) in recently diagnosed patients with BD.

METHODS

Eighty remitted patients with BD and 60 healthy controls (HC) underwent fMRI during which they performed a card guessing task. Of these, 41 patients and 36 HC were re-scanned after 16 months. We compared reward-related neural activity between groups at baseline and longitudinally and assessed the impact of mood relapse.

RESULTS

Patients showed lower RPE signal in areas of the ventrolateral prefrontal cortex (vlPFC) than HC. In these regions, the HC showed decrease in RPE signal over time, which was absent in patients. Patients further exhibited decreased EV signal in the occipital cortex across baseline and follow-up. Patients who remained in remission showed normalization of the EV signal at follow-up. Baseline activity in the identified regions was not associated with subsequent relapse.

LIMITATIONS

Follow-up scans were only available in a relatively small sample. Medication status, follow-up time and BD illness duration prior to diagnosis varied.

CONCLUSIONS

Lower RPE signal in the vlPFC in patients with BD at baseline and its lack of normative reduction over time may represent a trait marker of dysfunctional reward-based learning or habituation. The increase in EV signal in the occipital cortex over time in patients who remained in remission may indicate normalization of reward anticipation activity.

Amygdala response predicts clinical symptom reduction in patients with borderline personality disorder: A pilot fMRI study

Borderline personality disorder (BPD) is a prevalent, devastating, and heterogeneous psychiatric disorder. Treatment success is highly variable within this patient group. A cognitive neuroscientific approach to BPD might contribute to precision psychiatry by identifying neurocognitive factors that predict who will benefit from a specific treatment. Here, we build on observations that BPD is accompanied by the enhanced impact of the aversive effect on behavior and abnormal neural signaling in the amygdala. We assessed whether BPD is accompanied by abnormal aversive regulation of instrumental behavior and associated neural signaling, in a manner that is predictive of symptom reduction after therapy. We tested a clinical sample of 15 female patients with BPD, awaiting dialectical behavior therapy (DBT), and 16 matched healthy controls using fMRI and an aversive Pavlovian-to-instrumental transfer (PIT) task that assesses how instrumental behaviors are influenced by aversive Pavlovian stimuli. Patients were assessed 1 year after the start of DBT to quantify changes in BPD symptom severity. At baseline, behavioral aversive PIT and associated neural signaling did not differ between groups. However, the BOLD signal in the amygdala measured during aversive PIT was associated with symptom reduction at 1-year follow-up: higher PIT-related aversive amygdala signaling before treatment was associated with reduced clinical improvement at follow-up. Thus, within the evaluated group of BPD patients, the BOLD signal in the amygdala before treatment was related to clinical symptom reduction 1 year after the start of treatment. The results suggest that less PIT-related responsiveness of the amygdala increases the chances of treatment success. We note that the relatively small sample size is a limitation of this study and that replication is warranted.

Cross-paradigm integration shows a common neural basis for aversive and appetitive conditioning

Sharing imaging data and comparing them across different psychological tasks is becoming increasingly possible as the open science movement advances. Such cross-paradigm integration has the potential to identify commonalities in findings that neighboring areas of study thought to be paradigm-specific. However, even the integration of research from closely related paradigms, such as aversive and appetitive classical conditioning is rare - even though qualitative comparisons already hint at how similar the 'fear network' and 'reward network' may be. We aimed to validate these theories by taking a multivariate approach to assess commonalities across paradigms empirically. Specifically, we quantified the similarity of an aversive conditioning pattern derived from meta-analysis to appetitive conditioning fMRI data. We tested pattern expression in three independent appetitive conditioning studies with 29, 76 and 38 participants each. During fMRI scanning, participants in each cohorts performed an appetitive conditioning task in which a CS+ was repeatedly rewarded with money and a CS- was never rewarded. The aversive pattern was highly similar to appetitive CS+ > CS- contrast maps across samples and variations of the appetitive conditioning paradigms. Moreover, the pattern distinguished the CS+ from the CS- with above-chance accuracy in every sample. These findings provide robust empirical evidence for an underlying neural system common to appetitive and aversive learning. We believe that this approach provides a way to empirically integrate the steadily growing body of fMRI findings across paradigms.

Analysis of individual differences in neurofeedback training illuminates successful self-regulation of the dopaminergic midbrain

The dopaminergic midbrain is associated with reinforcement learning, motivation and decision-making – functions often disturbed in neuropsychiatric disorders. Previous research has shown that dopaminergic midbrain activity can be endogenously modulated via neurofeedback. However, the robustness of endogenous modulation, a requirement for clinical translation, is unclear. Here, we examine whether the activation of particular brain regions associates with successful regulation transfer when feedback is no longer available. Moreover, to elucidate mechanisms underlying effective self-regulation, we study the relation of successful transfer with learning (temporal difference coding) outside the midbrain during neurofeedback training and with individual reward sensitivity in a monetary incentive delay (MID) task. Fifty-nine participants underwent neurofeedback training either in standard (Study 1 N = 15, Study 2 N = 28) or control feedback group (Study 1, N = 16). We find that successful self-regulation is associated with prefrontal reward sensitivity in the MID task (N = 25), with a decreasing relation between prefrontal activity and midbrain learning signals during neurofeedback training and with increased activity within cognitive control areas during transfer. The association between midbrain self-regulation and prefrontal temporal difference and reward sensitivity suggests that reinforcement learning contributes to successful self-regulation. Our findings provide insights in the control of midbrain activity and may facilitate individually tailoring neurofeedback training.

Analysis of real-time fMRI data from 59 participants undergoing neurofeedback training suggests that reinforcement learning contributes to successful self-regulation in the dopaminergic midbrain.

Meta-analysis of human prediction error for incentives, perception, cognition, and action

Prediction errors (PEs) are a keystone for computational neuroscience. Their association with midbrain neural firing has been confirmed across species and has inspired the construction of artificial intelligence that can outperform humans. However, there is still much to learn. Here, we leverage the wealth of human PE data acquired in the functional neuroimaging setting in service of a deeper understanding, using an MKDA (multi-level kernel-based density) meta-analysis. Studies were identified with Google Scholar, and we included studies with healthy adult participants that reported activation coordinates corresponding to PEs published between 1999-2018. Across 264 PE studies that have focused on reward, punishment, action, cognition, and perception, consistent with domain-general theoretical models of prediction error we found midbrain PE signals during cognitive and reward learning tasks, and an insula PE signal for perceptual, social, cognitive, and reward prediction errors. There was evidence for domain-specific error signals--in the visual hierarchy during visual perception, and the dorsomedial prefrontal cortex during social inference. We assessed bias following prior neuroimaging meta-analyses and used family-wise error correction for multiple comparisons. This organization of computation by region will be invaluable in building and testing mechanistic models of cognitive function and dysfunction in machines, humans, and other animals. Limitations include small sample sizes and ROI masking in some included studies, which we addressed by weighting each study by sample size, and directly comparing whole brain vs. ROI-based results.

Learning under social versus nonsocial uncertainty: A meta-analytic approach

Much of the uncertainty that clouds our understanding of the world springs from the covert values and intentions held by other people. Thus, it is plausible that specialized mechanisms that compute learning signals under uncertainty of exclusively social origin operate in the brain. To test this hypothesis, we scoured academic databases for neuroimaging studies involving learning under uncertainty, and performed a meta‐analysis of brain activation maps that compared learning in the face of social versus nonsocial uncertainty. Although most of the brain activations associated with learning error signals were shared between social and nonsocial conditions, we found some evidence for functional segregation of error signals of exclusively social origin during learning in limited regions of ventrolateral prefrontal cortex and insula. This suggests that most behavioral adaptations to navigate social environments are reused from frontal and subcortical areas processing generic value representation and learning, but that a specialized circuitry might have evolved in prefrontal regions to deal with social context representation and strategic action.

Neural Encoding of Food and Monetary Reward Delivery

Different types of rewards such as food and money can similarly drive our behavior owing to shared brain processes encoding their subjective value. However, while the value of money is abstract and needs to be learned, the value of food is rooted in the innate processing of sensory properties and nutritional utilization. Yet, the actual consumption of food and the receipt of money have never been directly contrasted in the same experiment, questioning what unique neural processes differentiate those reward types. To fill this gap, we examined the distinct and common neural responses to the delivery of food and monetary rewards during fMRI. In a novel experimental approach, we parametrically manipulated the subjective value of food and monetary rewards by modulating the quantities of administered palatable milkshake and monetary gains. The receipt of increasing amounts of milkshake and money recruited the ventral striatum and the ventromedial prefrontal cortex, previously associated with value encoding. Notably, the consumption and the subsequent evaluation of increasing quantities of milkshake relative to money revealed an extended recruitment of brain regions related to taste, somatosensory processing, and salience. Moreover, we detected a decline of reward encoding in the ventral tegmental area, nucleus accumbens, and vmPFC, indicating that these regions may be susceptible to time-dependent effects upon accumulation of food and money rewards. Relative to monetary gains, the consumption and evaluation of palatable milkshakes engaged complex neural processing over and above value tracking, emphasizing the critical contribution of taste and other sensory properties to the processing of food rewards. Furthermore, our results highlight the need to closely monitor metabolic states and neural responses to the accumulation of rewards to pinpoint the mechanisms underlying time-dependent dynamics of reward-related processing.

Decision neuroscience and neuroeconomics: Recent progress and ongoing challenges

In the past decade, decision neuroscience and neuroeconomics have developed many new insights in the study of decision making. This review provides an overarching update on how the field has advanced in this time period. Although our initial review a decade ago outlined several theoretical, conceptual, methodological, empirical, and practical challenges, there has only been limited progress in resolving these challenges. We summarize significant trends in decision neuroscience through the lens of the challenges outlined for the field and review examples where the field has had significant, direct, and applicable impacts across economics and psychology. First, we review progress on topics including reward learning, explore-exploit decisions, risk and ambiguity, intertemporal choice, and valuation. Next, we assess the impacts of emotion, social rewards, and social context on decision making. Then, we follow up with how individual differences impact choices and new exciting developments in the prediction and neuroforecasting of future decisions. Finally, we consider how trends in decision-neuroscience research reflect progress toward resolving past challenges, discuss new and exciting applications of recent research, and identify new challenges for the field. This article is categorized under: Psychology > Reasoning and Decision Making Psychology > Emotion and Motivation.

Functional activation of insula and dorsal anterior cingulate for conflict control against larger monetary loss in young adults with subthreshold depression: a preliminary study

Subthreshold depression (StD) is associated with higher risk of later developing major depressive disorder (MDD). Deficits of goal-directed behaviors regarding the motional, motivational, and conflict control are found in MDD. The current study examined neural underpinning of conflict control against monetary punishment in StD compared to MDD and healthy controls (HC). Seventy-one participants (HC, n = 27; StD, n = 21; MDD, n = 23) in their mid-20's completed self-reports. Preprocessing of functional magnetic resonance imaging acquired for the Simon task against larger or smaller monetary punishment was conducted using ENIGMA HALFpipe version 1.2.1. Neural correlates of conflict control against monetary punishment that could vary with either diagnosis or PHQ-9 total score were examined using a general linear model of FSL. Simon effect was effective for reaction time and accuracy in every subgroup of diagnosis and regardless of the size of monetary punishment. Conflict control against larger monetary loss was associated with higher functional activation of left insula in StD than HC and MDD. StD showed lower functional activation of left dorsal anterior cingulate (dACC) than MDD for conflict control against larger monetary loss. For conflict control against smaller monetary loss, StD demonstrated higher functional activation of left paracentral lobule and right putamen compared to HC. Directed acyclic graphs showed directional associations from suicidal ideation, sadness, and concentration difficulty to functional activation of paracentral lobule, ventromedial prefrontal cortex (vmPFC), and thalamus for conflict control against monetary loss. Differential functional activation of insula and dACC for conflict control against larger monetary loss could be a brain phenotype of StD. Item-level depressive symptoms of suicidal ideation, sadness, and concentration difficulty could be reflected in the conflict control-related functional activation of paracentral lobule (against smaller monetary loss), vmPFC and thalamus (against larger monetary loss), respectively.

Motivational signals disrupt metacognitive signals in the human ventromedial prefrontal cortex

A growing body of evidence suggests that, during decision-making, BOLD signal in the ventromedial prefrontal cortex (VMPFC) correlates both with motivational variables – such as incentives and expected values – and metacognitive variables – such as confidence judgments – which reflect the subjective probability of being correct. At the behavioral level, we recently demonstrated that the value of monetary stakes bias confidence judgments, with gain (respectively loss) prospects increasing (respectively decreasing) confidence judgments, even for similar levels of difficulty and performance. If and how this value-confidence interaction is reflected in the VMPFC remains unknown. Here, we used an incentivized perceptual decision-making fMRI task that dissociates key decision-making variables, thereby allowing to test several hypotheses about the role of the VMPFC in the value-confidence interaction. While our initial analyses seemingly indicate that the VMPFC combines incentives and confidence to form an expected value signal, we falsified this conclusion with a meticulous dissection of qualitative activation patterns. Rather, our results show that strong VMPFC confidence signals observed in trials with gain prospects are disrupted in trials with no – or negative (loss) – monetary prospects. Deciphering how decision variables are represented and interact at finer scales seems necessary to better understand biased (meta)cognition.

The human ventromedial prefrontal cortex helps to determine value and confidence in certain decisions, but only in situations when there is a potential for a (monetary) reward.

Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making

Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.

Reinforcement Learning Disruptions in Individuals With Depression and Sensitivity to Symptom Change Following Cognitive Behavioral Therapy

IMPORTANCE

Major depressive disorder is prevalent and impairing. Parsing neurocomputational substrates of reinforcement learning in individuals with depression may facilitate a mechanistic understanding of the disorder and suggest new cognitive therapeutic targets.

OBJECTIVE

To determine associations among computational model-derived reinforcement learning parameters, depression symptoms, and symptom changes after treatment.

DESIGN, SETTING, AND PARTICIPANTS

In this mixed cross-sectional-cohort study, individuals performed reward and loss variants of a probabilistic learning task during functional magnetic resonance imaging at baseline and follow-up. A volunteer sample with and without a depression diagnosis was recruited from the community. Participants were assessed from July 2011 to February 2017, and data were analyzed from May 2017 to May 2021.

MAIN OUTCOMES AND MEASURES

Computational model-based analyses of participants' choices assessed a priori hypotheses about associations between components of reward-based and loss-based learning with depression symptoms. Changes in both learning parameters and symptoms were then assessed in a subset of participants who received cognitive behavioral therapy (CBT).

RESULTS

Of 101 included adults, 69 (68.3%) were female, and the mean (SD) age was 34.4 (11.2) years. A total of 69 participants with a depression diagnosis and 32 participants without a depression diagnosis were included at baseline; 48 participants (28 with depression who received CBT and 20 without depression) were included at follow-up (mean [SD] of 115.1 [15.6] days). Computational model-based analyses of behavioral choices and neural data identified associations of learning with symptoms during reward learning and loss learning, respectively. During reward learning only, anhedonia (and not negative affect or arousal) was associated with model-derived learning parameters (learning rate: posterior mean regression β = -0.14; 95% credible interval [CrI], -0.12 to -0.03; outcome sensitivity: posterior mean regression β = 0.18; 95% CrI, 0.02 to 0.37) and neural learning signals (moderation of association between striatal prediction error and expected value signals: t97 = -2.10; P = .04). During loss learning only, negative affect (and not anhedonia or arousal) was associated with learning parameters (outcome shift: posterior mean regression β = -0.11; 95% CrI, -0.20 to -0.01) and disrupted neural encoding of learning signals (association with subgenual anterior cingulate prediction error signals: r = -0.28; P = .005). Symptom improvement following CBT was associated with normalization of learning parameters that were disrupted at baseline (reward learning rate: posterior mean regression β = 0.15; 90% CrI, 0.001 to 0.41; loss outcome shift: posterior mean regression β = 0.42; 90% CrI, 0.09 to 0.77).

CONCLUSIONS AND RELEVANCE

In this study, the mapping of reinforcement learning components to symptoms of major depression revealed mechanistic features associated with these symptoms and points to possible learning-based therapeutic processes and targets.

Discordant Effects of Cannabinoid 2 Receptor Antagonism/Inverse Agonism During Adolescence on Pavlovian and Instrumental Reward Learning in Adult Male Rats

The endocannabinoid system is responsible for regulating a spectrum of physiological activities and plays a critical role in the developing brain. During adolescence, the endocannabinoid system is particularly sensitive to external insults that may change the brain’s developmental trajectory. Cannabinoid receptor type 2 (CB2R) was initially thought to predominantly function in the peripheral nervous system, but more recent studies have implicated its role in the mesolimbic pathway, a network largely attributed to reward circuitry and reward motivated behavior, which undergoes extensive changes during adolescence. It is therefore important to understand how CB2R modulation during adolescence can impact reward-related behaviors in adulthood. In this study, adolescent male rats (postnatal days 28–41) were exposed to a low or high dose of the CB2R antagonist/inverse agonist SR144528 and Pavlovian autoshaping and instrumental conditional behavioral outcomes were measured in adulthood. SR144528-treated rats had significantly slower acquisition of the autoshaping task, seen by less lever pressing behavior over time [F_{(2, 19)} = 5.964, p = 0.010]. Conversely, there was no effect of adolescent SR144528 exposure on instrumental conditioning. These results suggest that modulation of the CB2R in adolescence differentially impacts reward-learning behaviors in adulthood.

Neural Responses to Social Reward Predict Depressive Symptoms in Adolescent Girls During the COVID-19 Pandemic

OBJECTIVE

Adolescent depression is increasing during the COVID-19 pandemic, possibly related to dramatic social changes. Individual-level factors that contribute to social functioning, such as temperament and neural reactivity to social feedback, may confer risk for or resilience against depressive symptoms during the pandemic.

METHODS

Ninety-three girls (12-17 years) oversampled for high shy/fearful temperament were recruited from a longitudinal study for a follow-up COVID-19 study. During the parent study (2016-2018), participants completed a functional magnetic resonance imaging task eliciting neural activity to performance-related social feedback. Depressive symptoms were assessed during the parent study and COVID-19 follow-up (April-May 2020). In 65 participants with complete data, we examined how interactions between temperament and neural activation to social reward or punishment in a socio-affective brain network predict depressive symptoms during COVID-19.

RESULTS

Depressive symptoms increased during COVID-19. Significant interactions between temperament and caudate, putamen, and insula activation to social reward were found. Girls high in shy/fearful temperament showed negative associations between neural activation to social reward and COVID-19 depressive symptoms, whereas girls lower in shy/fearful temperament showed positive associations.

CONCLUSIONS

Girls high in shy/fearful temperament with reduced neural activation to social reward may be less likely to engage socially, which could be detrimental during the pandemic when social interactions are limited. In contrast, girls lower in shy/fearful temperament with heightened neural reactivity to social reward may be highly motivated to engage socially, which could also be detrimental with limited social opportunities. In both cases, improving social connection during the pandemic may attenuate or prevent depressive symptoms.

Multi-scale neural decoding and analysis

Objective.

Complex spatiotemporal neural activity encodes rich information related to behavior and cognition. Conventional research has focused on neural activity acquired using one of many different measurement modalities, each of which provides useful but incomplete assessment of the neural code. Multi-modal techniques can overcome tradeoffs in the spatial and temporal resolution of a single modality to reveal deeper and more comprehensive understanding of system-level neural mechanisms. Uncovering multi-scale dynamics is essential for a mechanistic understanding of brain function and for harnessing neuroscientific insights to develop more effective clinical treatment.

Approach.

We discuss conventional methodologies used for characterizing neural activity at different scales and review contemporary examples of how these approaches have been combined. Then we present our case for integrating activity across multiple scales to benefit from the combined strengths of each approach and elucidate a more holistic understanding of neural processes.

Main results.

We examine various combinations of neural activity at different scales and analytical techniques that can be used to integrate or illuminate information across scales, as well the technologies that enable such exciting studies. We conclude with challenges facing future multi-scale studies, and a discussion of the power and potential of these approaches.

Significance.

This roadmap will lead the readers toward a broad range of multi-scale neural decoding techniques and their benefits over single-modality analyses. This Review article highlights the importance of multi-scale analyses for systematically interrogating complex spatiotemporal mechanisms underlying cognition and behavior.

Disentangling predictive processing in the brain: a meta-analytic study in favour of a predictive network

According to the predictive coding (PC) theory, the brain is constantly engaged in predicting its upcoming states and refining these predictions through error signals. Despite extensive research investigating the neural bases of this theory, to date no previous study has systematically attempted to define the neural mechanisms of predictive coding across studies and sensory channels, focussing on functional connectivity. In this study, we employ a coordinate-based meta-analytical approach to address this issue. We first use the Activation Likelihood Estimation (ALE) algorithm to detect spatial convergence across studies, related to prediction error and encoding. Overall, our ALE results suggest the ultimate role of the left inferior frontal gyrus and left insula in both processes. Moreover, we employ a meta-analytic connectivity method (Seed-Voxel Correlations Consensus). This technique reveals a large, bilateral predictive network, which resembles large-scale networks involved in task-driven attention and execution. In sum, we find that: (i) predictive processing seems to occur more in certain brain regions than others, when considering different sensory modalities at a time; (ii) there is no evidence, at the network level, for a distinction between error and prediction processing.

Meta-analytic clustering dissociates brain activity and behavior profiles across reward processing paradigms

Reward learning is a ubiquitous cognitive mechanism guiding adaptive choices and behaviors, and when impaired, can lead to considerable mental health consequences. Reward-related functional neuroimaging studies have begun to implicate networks of brain regions essential for processing various peripheral influences (e.g., risk, subjective preference, delay, social context) involved in the multifaceted reward processing construct. To provide a more complete neurocognitive perspective on reward processing that synthesizes findings across the literature while also appreciating these peripheral influences, we used emerging meta-analytic techniques to elucidate brain regions, and in turn networks, consistently engaged in distinct aspects of reward processing. Using a data-driven, meta-analytic, k-means clustering approach, we dissociated seven meta-analytic groupings (MAGs) of neuroimaging results (i.e., brain activity maps) from 749 experimental contrasts across 176 reward processing studies involving 13,358 healthy participants. We then performed an exploratory functional decoding approach to gain insight into the putative functions associated with each MAG. We identified a seven-MAG clustering solution that represented dissociable patterns of convergent brain activity across reward processing tasks. Additionally, our functional decoding analyses revealed that each of these MAGs mapped onto discrete behavior profiles that suggested specialized roles in predicting value (MAG-1 & MAG-2) and processing a variety of emotional (MAG-3), external (MAG-4 & MAG-5), and internal (MAG-6 & MAG-7) influences across reward processing paradigms. These findings support and extend aspects of well-accepted reward learning theories and highlight large-scale brain network activity associated with distinct aspects of reward processing.

Search for solutions, learning, simulation, and choice processes in suicidal behavior

Suicide may be viewed as an unfortunate outcome of failures in decision processes. Such failures occur when the demands of a crisis exceed a person's capacity to (i) search for options, (ii) learn and simulate possible futures, and (iii) make advantageous value‐based choices. Can individual‐level decision deficits and biases drive the progression of the suicidal crisis? Our overview of the evidence on this question is informed by clinical theory and grounded in reinforcement learning and behavioral economics. Cohort and case–control studies provide strong evidence that limited cognitive capacity and particularly impaired cognitive control are associated with suicidal behavior, imposing cognitive constraints on decision‐making. We conceptualize suicidal ideation as an element of impoverished consideration sets resulting from a search for solutions under cognitive constraints and mood‐congruent Pavlovian influences, a view supported by mostly indirect evidence. More compelling is the evidence of impaired learning in people with a history of suicidal behavior. We speculate that an inability to simulate alternative futures using one's model of the world may undermine alternative solutions in a suicidal crisis. The hypothesis supported by the strongest evidence is that the selection of suicide over alternatives is facilitated by a choice process undermined by randomness. Case–control studies using gambling tasks, armed bandits, and delay discounting support this claim. Future experimental studies will need to uncover real‐time dynamics of choice processes in suicidal people. In summary, the decision process framework sheds light on neurocognitive mechanisms that facilitate the progression of the suicidal crisis.

This article is categorized under:

Economics > Individual Decision‐Making

Psychology > Emotion and Motivation

Psychology > Learning

Neuroscience > Behavior

Robust and distributed neural representation of action values

Studies in rats, monkeys, and humans have found action-value signals in multiple regions of the brain. These findings suggest that action-value signals encoded in these brain structures bias choices toward higher expected rewards. However, previous estimates of action-value signals might have been inflated by serial correlations in neural activity and also by activity related to other decision variables. Here, we applied several statistical tests based on permutation and surrogate data to analyze neural activity recorded from the striatum, frontal cortex, and hippocampus. The results show that previously identified action-value signals in these brain areas cannot be entirely accounted for by concurrent serial correlations in neural activity and action value. We also found that neural activity related to action value is intermixed with signals related to other decision variables. Our findings provide strong evidence for broadly distributed neural signals related to action value throughout the brain.

Reward and fictive prediction error signals in ventral striatum: asymmetry between factual and counterfactual processing

Reward prediction error, the difference between the expected and obtained reward, is known to act as a reinforcement learning neural signal. In the current study, we propose a model fitting approach that combines behavioral and neural data to fit computational models of reinforcement learning. Briefly, we penalized subject-specific fitted parameters that moved away too far from the group median, except when that deviation led to an improvement in the model's fit to neural responses. By means of a probabilistic monetary learning task and fMRI, we compared our approach with standard model fitting methods. Q-learning outperformed actor-critic at both behavioral and neural level, although the inclusion of neuroimaging data into model fitting improved the fit of actor-critic models. We observed both action-value and state-value prediction error signals in the striatum, while standard model fitting approaches failed to capture state-value signals. Finally, left ventral striatum correlated with reward prediction error while right ventral striatum with fictive prediction error, suggesting a functional hemispheric asymmetry regarding prediction-error driven learning.

Abnormal reward valuation and event-related connectivity in unmedicated major depressive disorder

BACKGROUND

Experience of emotion is closely linked to valuation. Mood can be viewed as a bias to experience positive or negative emotions and abnormally biased subjective reward valuation and cognitions are core characteristics of major depression.

METHODS

Thirty-four unmedicated subjects with major depressive disorder and controls estimated the probability that fractal stimuli were associated with reward, based on passive observations, so they could subsequently choose the higher of either their estimated fractal value or an explicitly presented reward probability. Using model-based functional magnetic resonance imaging, we estimated each subject's internal value estimation, with psychophysiological interaction analysis used to examine event-related connectivity, testing hypotheses of abnormal reward valuation and cingulate connectivity in depression.

RESULTS

Reward value encoding in the hippocampus and rostral anterior cingulate was abnormal in depression. In addition, abnormal decision-making in depression was associated with increased anterior mid-cingulate activity and a signal in this region encoded the difference between the values of the two options. This localised decision-making and its impairment to the anterior mid-cingulate cortex (aMCC) consistent with theories of cognitive control. Notably, subjects with depression had significantly decreased event-related connectivity between the aMCC and rostral cingulate regions during decision-making, implying impaired communication between the neural substrates of expected value estimation and decision-making in depression.

CONCLUSIONS

Our findings support the theory that abnormal neural reward valuation plays a central role in major depressive disorder (MDD). To the extent that emotion reflects valuation, abnormal valuation could explain abnormal emotional experience in MDD, reflect a core pathophysiological process and be a target of treatment.

Unraveling the Temporal Dynamics of Reward Signals in Music-Induced Pleasure with TMS

Music's ability to induce feelings of pleasure has been the subject of intense neuroscientific research lately. Prior neuroimaging studies have shown that music-induced pleasure engages cortico-striatal circuits related to the anticipation and receipt of biologically relevant rewards/incentives, but these reports are necessarily correlational. Here, we studied both the causal role of this circuitry and its temporal dynamics by applying transcranial magnetic stimulation (TMS) over the left dorsolateral PFC combined with fMRI in 17 male and female participants. Behaviorally, we found that, in accord with previous findings, excitation of fronto-striatal pathways enhanced subjective reports of music-induced pleasure and motivation, whereas inhibition of the same circuitry led to the reduction of both. fMRI activity patterns indicated that these behavioral changes were driven by bidirectional TMS-induced alteration of fronto-striatal function. Specifically, changes in activity in the NAcc predicted modulation of both hedonic and motivational responses, with a dissociation between pre-experiential versus experiential components of musical reward. In addition, TMS-induced changes in the fMRI functional connectivity between the NAcc and frontal and auditory cortices predicted the degree of modulation of hedonic responses. These results indicate that the engagement of cortico-striatal pathways and the NAcc, in particular, is indispensable to experience rewarding feelings from music.SIGNIFICANCE STATEMENT Neuroimaging studies have shown that music-induced pleasure engages cortico-striatal circuits involved in the processing of biologically relevant rewards. Yet, these reports are necessarily correlational. Here, we studied both the causal role of this circuitry and its temporal dynamics by combining brain stimulation over the frontal cortex with functional imaging. Behaviorally, we found that excitation and inhibition of fronto-striatal pathways enhanced and disrupted, respectively, subjective reports of music-induced pleasure and motivation. These changes were associated with changes in NAcc activity and NAcc coupling with frontal and auditory cortices, dissociating between pre-experimental versus experiential components of musical reward. These results indicate that the engagement of cortico-striatal pathways, and the NAcc in particular, is indispensable to experience rewarding feeling from music.

The Prisoner's Dilemma paradigm provides a neurobiological framework for the social decision cascade

To function during social interactions, we must be able to consider and coordinate our actions with other people's perspectives. This process unfolds from decision-making, to anticipation of that decision's consequences, to feedback about those consequences, in what can be described as a "cascade" of three phases. The iterated Prisoner's Dilemma (iPD) task, an economic-exchange game used to illustrate how people achieve stable cooperation over repeated interactions, provides a framework for examining this "social decision cascade". In the present study, we examined neural activity associated with the three phases of the cascade, which can be isolated during iPD game rounds. While undergoing functional magnetic resonance imaging (fMRI), 31 adult participants made a) decisions about whether to cooperate with a co-player for a monetary reward, b) anticipated the co-player's decision, and then c) learned the co-player's decision. Across all three phases, participants recruited the temporoparietal junction (TPJ) and the dorsomedial prefrontal cortex (dmPFC), regions implicated in numerous facets of social reasoning such as perspective-taking and the judgement of intentions. Additionally, a common distributed neural network underlies both decision-making and feedback appraisal; however, differences were identified in the magnitude of recruitment between both phases. Furthermore, there was limited evidence that anticipation following the decision to defect evoked a neural signature that is distinct from the signature of anticipation following the decision to cooperate. This study is the first to delineate the neural substrates of the entire social decision cascade in the context of the iPD game.

Instrumental and Pavlovian Mechanisms in Alcohol Use Disorder

Purpose of Review

Current theories of alcohol use disorders (AUD) highlight the importance of Pavlovian and instrumental learning processes mainly based on preclinical animal studies. Here, we summarize available evidence for alterations of those processes in human participants with AUD with a focus on habitual versus goal-directed instrumental learning, Pavlovian conditioning, and Pavlovian-to-instrumental transfer (PIT) paradigms.

Recent Findings

The balance between habitual and goal-directed control in AUD participants has been studied using outcome devaluation or sequential decision-making procedures, which have found some evidence of reduced goal-directed/model-based control, but little evidence for stronger habitual responding. The employed Pavlovian learning and PIT paradigms have shown considerable differences regarding experimental procedures, e.g., alcohol-related or conventional reinforcers or stimuli.

Summary

While studies of basic learning processes in human participants with AUD support a role of Pavlovian and instrumental learning mechanisms in the development and maintenance of drug addiction, current studies are characterized by large variability regarding methodology, sample characteristics, and results, and translation from animal paradigms to human research remains challenging. Longitudinal approaches with reliable and ecologically valid paradigms of Pavlovian and instrumental processes, including alcohol-related cues and outcomes, are warranted and should be combined with state-of-the-art imaging techniques, computational approaches, and ecological momentary assessment methods.

Sex differences in neural responses to reward and the influences of individual reward and punishment sensitivity

BACKGROUND

Men and women show differences in sensitivity to reward and punishment, which may impact behavior in health and disease. However, the neural bases of these sex differences remain under-investigated. Here, by combining functional magnetic resonance imaging (fMRI) and a variant of the Monetary Incentive Delay Task (MIDT), we examined sex differences in the neural responses to wins and losses and how individual reward and punishment sensitivity modulates these regional activities.

METHODS

Thirty-sex men and 27 women participated in the fMRI study. We assessed sensitivity to punishment (SP) and sensitivity to reward (SR) with the Sensitivity to Punishment and Sensitivity to Reward Questionnaire (SPSRQ). In the MIDT, participants pressed a button to collect reward ($1, 1¢, or nil), with the reaction time window titrated across trials so participants achieved a success rate of approximately 67%. We processed the Imaging data with published routines and evaluated the results with a corrected threshold.

RESULTS

Women showed higher SP score than men and men showed higher SR score than women. Men relative to women showed higher response to the receipt of dollar or cent reward in bilateral orbitofrontal and visual cortex. Men as compared to women also showed higher response to dollar loss in bilateral orbitofrontal cortex. Further, in whole-brain regressions, women relative to men demonstrated more significant modulation by SP in the neural responses to wins and larger wins, and the sex differences were confirmed by slope tests.

CONCLUSIONS

Together, men showed higher SR and neural sensitivity to both wins, large or small, and losses than women. Individual differences in SP were associated with diminished neural responses to wins and larger wins in women only. These findings highlight how men and women may differ in reward-related brain activations in the MIDT and add to the imaging literature of sex differences in cognitive and affective functions.

Understanding the Neurocomputational Mechanisms of Antidepressant Placebo Effects

Over the last two decades, neuroscientists have used antidepressant placebo probes to examine the biological mechanisms implicated in antidepressant placebo effects. However, findings from these studies have not yet elucidated a model-based theory that would explain the mechanism through which antidepressant expectancies evolve to induce persistent mood changes. Emerging evidence suggests that antidepressant placebo effects may be informed by models of reinforcement learning (RL). Such that an individual's expectation of improvement is updated with the arrival of new sensory evidence, by incorporating a reward prediction error (RPE), which signals the mismatch between the expected (expected value) and perceived improvement. Consistent with this framework, neuroimaging studies of antidepressant placebo effects have demonstrated placebo-induced μ-opioid activation and increased blood-oxygen-level dependent (BOLD) responses in regions tracking expected values (e.g., ventromedial prefrontal cortex (vmPFC)) and RPEs (e.g., ventral striatum (VS)). In this study, we will demonstrate the causal contribution of reward learning signals (expected values and RPEs) to antidepressant placebo effects by experimentally manipulating expected values using transcranial magnetic stimulation (TMS) targeting the vmPFC and μ-opioid striatal RPE signal using pharmacological approaches. We hypothesized that antidepressant placebo expectancies are represented in the vmPFC (expected value) and updated by means of μ-opioid-modulated striatal learning signal. In a 3 × 3 factorial double-blind design, we will randomize 120 antidepressant-free individuals with depressive symptoms to one of three between-subject opioid conditions: the μ-opioid agonist buprenorphine, the μ-opioid antagonist naltrexone, or an inert pill. Within each arm, individuals will be assigned to receive three within-subject counterbalanced forms of TMS targeting the vmPFC-intermittent Theta Burst Stimulation (TBS) expected to potentiate the vmPFC, continuous TBS expected to de-potentiate the vmPFC, or sham TBS. These experimental manipulations will be used to modulate trial-by-trial reward learning signals and related brain activity during the Antidepressant Placebo functional MRI (fMRI) Task to address the following aims: (1) investigate the relationship between reward learning signals within the vmPFC-VS circuit and antidepressant placebo effects; (2) examine the causal contribution of vmPFC expected value computations to antidepressant placebo effects; and (3) investigate the causal contribution of μ-opioid-modulated striatal RPEs to antidepressant placebo effects. The proposed study will be the first to investigate the causal contribution of μ-opioid-modulated vmPFC-VS learning signals to antidepressant placebo responses, paving the way for developing novel treatments modulating learning processes and objective means of quantifying and potentially reducing placebo effects during drug development.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT04276259.

A ventral striatal prediction error signal in human fear extinction learning

Animal studies have shown that the prediction error (PE) signal that drives fear extinction learning is encoded by phasic activity of midbrain dopamine (DA) neurons. Thus, the extinction PE resembles the appetitive PE that drives reward learning. In humans, fear extinction learning is less well understood. Using computational neuroimaging, a previous study from our group reported hemodynamic activity in the left ventral putamen, a subregion of the ventral striatum (VS), to correlate with a PE function derived from a formal associative learning model. The activity was modulated by genetic variation in a DA-related gene. To conceptually replicate and extend this finding, we here asked whether an extinction PE (EPE) signal in the left ventral putamen can also be observed when genotype information is not taken into account. Using an optimized experimental design for model estimation, we again observed EPE-related activity in the same striatal region, indicating that activation of this region is a feature of human extinction learning. We further observed significant EPE signals across wider parts of the VS as well as in frontal cortical areas. These results may suggest that the prediction errors during extinction learning are available to larger parts of the brain, as has also been observed in human neuroimaging studies of reward PE signaling. Conclusive evidence that the human EPE signal is of DAergic nature is still outstanding.

Common and distinct neural correlates of music and food-induced pleasure: A coordinate-based meta-analysis of neuroimaging studies

Neuroimaging studies have shown that, despite the abstractness of music, it may mimic biologically rewarding stimuli (e.g., food) in its ability to engage the brain's reward circuitry. However, due to the lack of research comparing music and other types of reward, it is unclear to what extent the recruitment of reward-related structures overlaps among domains. To achieve this goal, we performed a coordinate-based meta-analysis of 38 neuroimaging studies (703 subjects) comparing the brain responses specifically to music and food-induced pleasure. Both engaged a common set of brain regions, including the ventromedial prefrontal cortex, ventral striatum, and insula. Yet, comparative analyses indicated a partial dissociation in the engagement of the reward circuitry as a function of the type of reward, as well as additional reward type-specific activations in brain regions related to perception, sensory processing, and learning. These results support the idea that hedonic reactions rely on the engagement of a common reward network, yet through specific routes of access depending on the modality and nature of the reward.

Abnormal prefrontal cortex processing of reward prediction errors in recently diagnosed patients with bipolar disorder and their unaffected relatives

OBJECTIVE

Bipolar disorder (BD) has been associated with abnormal reward functioning including pleasure-seeking and impulsivity. Here we sought to clarify whether these changes can be attributed to abnormalities in the neural processing of reward valuation or error prediction. Moreover, we tested whether abnormalities in these processes are associated with familial vulnerability to BD.

METHODS

We obtained functional magnetic resonance imaging data from patients with recently diagnosed BD (n = 85), their unaffected first-degree relatives (n = 44), and healthy control participants (n = 66) while they were performing a monetary card game. We used a region-of-interest approach to test for group differences in the activation of the midbrain, the ventral striatum, and the prefrontal cortex during reward valuation and error prediction.

RESULTS

Patients with BD showed decreased prediction error signal in ventrolateral prefrontal cortex and the unaffected relatives showed decreased prediction error signal in the supplementary motor area in comparison to healthy controls. There were no significant group differences in the activation of the ventral striatum during the task. In healthy controls, prediction error signal in dorsal anterior cingulate cortex correlated with an out-of-scanner measure of motor inhibition but this association was absent in patients and relatives.

CONCLUSIONS

The findings indicate that abnormal reward processing in BD is primarily related to deficits in the engagement of prefrontal regions involved in inhibitory control during error prediction. In contrast, deficient activation in supplementary motor cortex involved in planning of movement emerged as a familial vulnerability to BD.

Differential reinforcement encoding along the hippocampal long axis helps resolve the explore-exploit dilemma

When making decisions, should one exploit known good options or explore potentially better alternatives? Exploration of spatially unstructured options depends on the neocortex, striatum, and amygdala. In natural environments, however, better options often cluster together, forming structured value distributions. The hippocampus binds reward information into allocentric cognitive maps to support navigation and foraging in such spaces. Here we report that human posterior hippocampus (PH) invigorates exploration while anterior hippocampus (AH) supports the transition to exploitation on a reinforcement learning task with a spatially structured reward function. These dynamics depend on differential reinforcement representations in the PH and AH. Whereas local reward prediction error signals are early and phasic in the PH tail, global value maximum signals are delayed and sustained in the AH body. AH compresses reinforcement information across episodes, updating the location and prominence of the value maximum and displaying goal cell-like ramping activity when navigating toward it.

The neural network underpinning social feedback contingent upon one's action: An fMRI study

Praise enhances motor performance; however, the underlying feedback pathway is unknown. Here, we hypothesized that the social evaluation feedback to the motor system is modified by the top-down effect of the social contingency valuation system, such as the anterior rostral medial prefrontal cortex (arMPFC). We developed a pseudo-interactive task that simplified a conversational student-teacher interaction and conducted a functional magnetic resonance imaging study with 33 participants (13 men, 20 women; mean age = 21.7 years; standard deviation = 2.0 years). The participant inside the scanner uttered the pseudo-English word to the English teacher outside the scanner. The teacher provided feedback of acceptance or rejection by either gestures or words, through video. As a control condition, the pseudo-word was read aloud by a computer. Approval from the teacher enhanced the participants' pleasure rate. Feedback to the participants' utterance, either rejection or acceptance, activated the arMPFC. Irrespective of the preceding utterance by self or computer, acceptance compared with rejection activated the right primary visual cortex (V1), and the reverse activated the left V1. This valence-dependent laterality of V1 activation indicates that the effect is not the domain-general modulation of visual processing. Instead, the early visual cortices are part of the valence-specific representation of the social signal. Physio-physiological interaction analysis with the seed regions in the right and left V1 and the modulator region in the arMPFC showed enhanced connectivity with the bilateral primary motor cortex. These findings indicate that the socially contingent, self-relevant signals from others act as feedback to the motor control system, and this process is mediated by the early visual cortex.