Temporal difference error prediction signal dysregulation in cocaine dependence

Effects of psychedelic, DOI, on nucleus accumbens dopamine signaling to predictable rewards and cues in rats

Psychedelics produce lasting therapeutic responses in neuropsychiatric diseases suggesting they may disrupt entrenched associations and catalyze learning. Here, we examine psychedelic 5-HT2A/2C agonist, DOI, effects on dopamine signaling in the nucleus accumbens (NAc) core, a region extensively linked to reward learning, motivation, and drug-seeking. We measure phasic dopamine transients following acute DOI administration in rats during well learned Pavlovian tasks in which sequential cues predict rewards. We find that DOI (0.0-1.2 mg/kg, i.p.) increases dopamine signals, photometrically measured using GRABDA optical sensor, to rewards and proximal reward cues, but not to the distal cues that predict these events. We determine that the elevated dopamine produced by DOI to reward cues occurs independently of DOI-induced changes in reward value. The increased dopamine associated with predictable reward cues and rewards supports DOI-induced increases in prediction error signaling. These findings lay a foundation for developing psychedelic strategies aimed at engaging error-driven learning mechanisms to disrupt entrenched associations or produce new associations.

A meta-analysis of electrophysiological biomarkers of reward and error monitoring in substance misuse

Substance use disorders are characterized by marked changes in reward and error processing. The primary objective of this meta-analysis was to estimate effect sizes for the reward positivity (RewP) and error-related negativity (ERN), two event-related potential indicators of outcome monitoring, in substance users compared to controls. The secondary objective was to test for moderation by demographic, substance type, and EEG experiment parameters. Final PubMed searches were performed in August 2023. Inclusion criteria were substance use disorder/dependence or validated self-report of substance misuse, RewP/ERN means available, healthy control comparison group, non-acute drug study, peer-reviewed journal, English language, and human participants. Selection bias was tested through modified Egger's regression and exploratory 3-parameter selection model tests. The RewP results (19 studies, 1641 participants) did not support an overall effect (Hedges' g = 0.07, 95% CI [-0.44, 0.58], p = .777) and nor effect of any moderators. The ERN results (20 studies, 1022 participants) indicated no significant overall effect (g = 0.41, 95%CI [-0.05, 0.88]). Subgroup analyses indicated that cocaine users had a blunted ERN compared to controls (g = 1.12, 95%CI [0.77, 1.47]). There was limited evidence for publication/small study bias. Although the results indicate a potential dissociation between substance types, this meta-analysis revealed the need for additional research on the RewP/ERN in substance using populations and for better designed experiments that adequately address research questions.

A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI

Introduction

Computational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses.

Methods

Simulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses.

Results

Initial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise.

Conclusion

Overall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder.

Reduced neural encoding of utility prediction errors in cocaine addiction

Influential accounts of addiction posit alterations in adaptive behavior driven by deficient dopaminergic prediction errors (PEs), signaling the discrepancy between actual and expected reward. Dopamine neurons encode these error signals in subjective terms, calibrated by individual risk preferences, as "utility" PEs. It remains unclear, however, whether people with drug addiction have PE deficits or their computational source. Here, using an analogous task to prior single-unit studies with known expectancies, we show that fMRI-measured PEs similarly reflect utility PEs. Relative to control participants, people with chronic cocaine addiction demonstrate reduced utility PEs in the dopaminoceptive ventral striatum, with similar trends in orbitofrontal cortex. Dissecting this PE signal into its subcomponent terms attributed these reductions to weaker striatal responses to received reward/utility, whereas suppression of activity with reward expectation was unchanged. These findings support that addiction may fundamentally disrupt PE signaling and reveal an underappreciated role for perceived reward value in this mechanism.

Reward prediction-errors weighted by cue salience produces addictive behaviours in simulations, with asymmetrical learning and steeper delay discounting

Dysfunction in learning and motivational systems are thought to contribute to addictive behaviours. Previous models have suggested that dopaminergic roles in learning and motivation could produce addictive behaviours through pharmacological manipulations that provide excess dopaminergic signalling towards these learning and motivational systems. Redish (2004) suggested a role based on dopaminergic signals of value prediction error, while (Zhang et al., 2009) suggested a role based on dopaminergic signals of motivation. However, both models present significant limitations. They do not explain the reduced sensitivity to drug-related costs/negative consequences, the increased impulsivity generally found in people with a substance use disorder, craving behaviours, and non-pharmacological dependence, all of which are key hallmarks of addictive behaviours. Here, we propose a novel mathematical definition of salience, that combines aspects of dopamine's role in both learning and motivation within the reinforcement learning framework. Using a single parameter regime, we simulated addictive behaviours that the (Zhang et al., 2009; Redish, 2004) models also produce but we went further in simulating the downweighting of drug-related negative prediction-errors, steeper delay discounting of drug rewards, craving behaviours and aspects of behavioural/non-pharmacological addictions. The current salience model builds on our recently proposed conceptual theory that salience modulates internal representation updating and may contribute to addictive behaviours by producing misaligned internal representations (Kalhan et al., 2021). Critically, our current mathematical model of salience argues that the seemingly disparate learning and motivational aspects of dopaminergic functioning may interact through a salience mechanism that modulates internal representation updating.

Blunted reward prediction error signals in internet gaming disorder

BACKGROUND

Internet gaming disorder (IGD) is a type of behavioural addictions. One of the key features of addiction is the excessive exposure to addictive objectives (e.g. drugs) reduces the sensitivity of the brain reward system to daily rewards (e.g. money). This is thought to be mediated via the signals expressed as dopaminergic reward prediction error (RPE). Emerging evidence highlights blunted RPE signals in drug addictions. However, no study has examined whether IGD also involves alterations in RPE signals that are observed in other types of addictions.

METHODS

To fill this gap, we used functional magnetic resonance imaging data from 45 IGD and 42 healthy controls (HCs) during a reward-related prediction-error task and utilised a psychophysiological interaction (PPI) analysis to characterise the underlying neural correlates of RPE and related functional connectivity.

RESULTS

Relative to HCs, IGD individuals showed impaired reinforcement learning, blunted RPE signals in multiple regions of the brain reward system, including the right caudate, left orbitofrontal cortex (OFC), and right dorsolateral prefrontal cortex (DLPFC). Moreover, the PPI analysis revealed a pattern of hyperconnectivity between the right caudate, right putamen, bilateral DLPFC, and right dorsal anterior cingulate cortex (dACC) in the IGD group. Finally, linear regression suggested that the connection between the right DLPFC and right dACC could significantly predict the variation of RPE signals in the left OFC.

CONCLUSIONS

These results highlight disrupted RPE signalling and hyperconnectivity between regions of the brain reward system in IGD. Reinforcement learning deficits may be crucial underlying characteristics of IGD pathophysiology.

Computational theory-driven studies of reinforcement learning and decision-making in addiction: What have we learned?

Computational psychiatry provides a powerful new approach for linking the behavioral manifestations of addiction to their precise cognitive and neurobiological substrates. However, this emerging area of research is still limited in important ways. While research has identified features of reinforcement learning and decision-making in substance users that differ from health, less emphasis has been placed on capturing addiction cycles/states dynamically, within-person. In addition, the focus on few behavioral variables at a time has precluded more detailed consideration of related processes and heterogeneous clinical profiles. We propose that a longitudinal and multidimensional examination of value-based processes, a type of dynamic "computational fingerprint", will provide a more complete understanding of addiction as well as aid in developing better tailored and timed interventions.

Computational models of drug use and addiction: A review

In this brief review, we describe current computational models of drug-use and addiction that fall into 2 broad categories: mathematically based models that rely on computational theories, and brain-based models that link computations to brain areas or circuits. Across categories, many are models of learning and decision-making, which may be compromised in addiction. Several mathematical models take predictive coding approaches, focusing on Bayesian prediction error. Other models focus on learning processes and (traditional) prediction error. Brain-based models have incorporated prefrontal cortex, basal ganglia, and the dopamine system, based on the effects of drugs on dopamine, motivation, and executive control circuits. Several models specifically describe how behavioral control may transition from habitual to goal-directed systems, consistent with computational accounts of compromised "model-based" control. Some brain-based models have linked this to the transition of behavioral control from ventral to dorsal striatum. Overall, we propose that while computational models capture some aspects of addiction and have advanced our thinking, most have focused on the effects of drug use rather than addiction per se, most have not been tested on and/or supported by human data, and few capture multiple stages and symptoms of addiction. We conclude by suggesting a path forward for computational models of addiction. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Neuroimaging reward, craving, learning, and cognitive control in substance use disorders: review and implications for treatment

Substance use disorder is a leading causes of preventable disease and mortality. Drugs of abuse cause molecular and cellular changes in specific brain regions and these neuroplastic changes are thought to play a role in the transition to uncontrolled drug use. Neuroimaging has identified neural substrates associated with problematic substance use and may offer clues to reduce its burden on the patient and society. Here, we provide a narrative review of neuroimaging studies that have examined the structures and circuits associated with reward, cues and craving, learning, and cognitive control in substance use disorders. Most studies use advanced MRI or positron emission tomography (PET). Many studies have focused on the dopamine neurons of the ventral tegmental area, and the regions where these neurons terminate, such as the striatum and prefrontal cortex. Decreases in dopamine receptors and transmission have been found in chronic users of drugs, alcohol, and nicotine. Recent studies also show evidence of differences in structure and function in substance users relative to controls in brain regions involved in salience evaluation, such as the insula and anterior cingulate cortex. Balancing between reward-related bottom-up and cognitive-control-related top-down processes is discussed in the context of neuromodulation as a potential treatment. Finally, some of the challenges for understanding substance use disorder using neuroimaging methods are discussed.

In Cocaine Dependence, Neural Prediction Errors During Loss Avoidance Are Increased With Cocaine Deprivation and Predict Drug Use

BACKGROUND

In substance-dependent individuals, drug deprivation and drug use trigger divergent behavioral responses to environmental cues. These divergent responses are consonant with data showing that short- and long-term adaptations in dopamine signaling are similarly sensitive to state of drug use. The literature suggests a drug state-dependent role of learning in maintaining substance use; evidence linking dopamine to both reinforcement learning and addiction provides a framework to test this possibility.

METHODS

In a randomized crossover design, 22 participants with current cocaine use disorder completed a probabilistic loss-learning task during functional magnetic resonance imaging while on and off cocaine (44 sessions). Another 54 participants without Axis I psychopathology served as a secondary reference group. Within-drug state and paired-subjects' learning effects were assessed with computational model-derived individual learning parameters. Model-based neuroimaging analyses evaluated effects of drug use state on neural learning signals. Relationships among model-derived behavioral learning rates (α+, α-), neural prediction error signals (δ+, δ-), cocaine use, and desire to use were assessed.

RESULTS

During cocaine deprivation, cocaine-dependent individuals exhibited heightened positive learning rates (α+), heightened neural positive prediction error (δ+) responses, and heightened association of α+ with neural δ+ responses. The deprivation-enhanced neural learning signals were specific to successful loss avoidance, comparable to participants without psychiatric conditions, and mediated a relationship between chronicity of drug use and desire to use cocaine.

CONCLUSIONS

Neurocomputational learning signals are sensitive to drug use status and suggest that heightened reinforcement by successful avoidance of negative outcomes may contribute to drug seeking during deprivation. More generally, attention to drug use state is important for delineating substrates of addiction.

Response inhibition and fronto-striatal-thalamic circuit dysfunction in cocaine addiction

BACKGROUND

Many studies have investigated how cognitive control may be compromised in cocaine addiction. Here, we extend this literature by employing spatial Independent Component Analysis (ICA) to describe circuit dysfunction in relation to impairment in response inhibition in cocaine addiction.

METHODS

Fifty-five cocaine-dependent (CD) and 55 age- and sex-matched non-drug-using healthy control individuals (HC) participated in the study. Task-relatedness of 40 independent components (ICs) was assessed using multiple regression analyses of component time courses with the modeled time courses of hemodynamic activity convolved with go success (GS), stop success (SS) and stop error (SE). This procedure produced beta-weights that represented the degree to which each IC was temporally associated with, or 'engaged', by each task event.

RESULTS

Behaviorally, CD participants showed prolonged stop signal reaction times (SSRTs) as compared to HC participants (p < 0.01). ICA identified two networks that showed differences in engagement related to SS between CD and HC (p < 0.05, FDR-corrected). The activity of the fronto-striatal-thalamic network was negatively correlated with SSRTs in HC but not in CD, suggesting a specific role of this network in mediating deficits of response inhibition in CD individuals. In contrast, the engagement of the fronto-parietal-temporal network did not relate to SSRTs, was similarly less engaged for both SS and SE trials, and may reflect attentional dysfunction in cocaine addiction.

CONCLUSIONS

This study highlights the utility of ICA in identifying neural circuitry engagement related to SST performance and suggests that specific networks may represent important targets in remedying executive-control impairment in cocaine addiction.

A Focus on Reward Prediction and the Lateral Habenula: Functional Alterations and the Behavioral Outcomes Induced by Drugs of Abuse

The lateral habenula (LHb) regulates reward learning and controls the updating of reward-related information. Drugs of abuse have the capacity to hijack the cellular and neurocircuit mechanisms mediating reward learning, forming non-adaptable, compulsive behaviors geared toward obtaining illicit substances. Here, we discuss current findings demonstrating how drugs of abuse alter intrinsic and synaptic LHb neuronal function. Additionally, we discuss evidence for how drug-induced LHb alterations may affect the ability to predict reward, potentially facilitating an addiction-like state. Altogether, we combine ex vivo and in vivo results for an overview of how drugs of abuse alter LHb function and how these functional alterations affect the ability to learn and update behavioral responses to hedonic external stimuli.

Reward Prediction Errors in Drug Addiction and Parkinson's Disease: from Neurophysiology to Neuroimaging

PURPOSE OF REVIEW

Surprises are important sources of learning. Cognitive scientists often refer to surprises as "reward prediction errors," a parameter that captures discrepancies between expectations and actual outcomes. Here, we integrate neurophysiological and functional magnetic resonance imaging (fMRI) results addressing the processing of reward prediction errors and how they might be altered in drug addiction and Parkinson's disease.

RECENT FINDINGS

By increasing phasic dopamine responses, drugs might accentuate prediction error signals, causing increases in fMRI activity in mesolimbic areas in response to drugs. Chronic substance dependence, by contrast, has been linked with compromised dopaminergic function, which might be associated with blunted fMRI responses to pleasant non-drug stimuli in mesocorticolimbic areas. In Parkinson's disease, dopamine replacement therapies seem to induce impairments in learning from negative outcomes. The present review provides a holistic overview of reward prediction errors across different pathologies and might inform future clinical strategies targeting impulsive/compulsive disorders.

Reduced activity in functional networks during reward processing is modulated by abstinence in cocaine addicts

Cocaine addiction is characterized by alterations in motivational and cognitive processes. Recent studies have shown that some alterations present in cocaine users may be related to the activity of large functional networks. The aim of this study was to investigate how these functional networks are modulated by non-drug rewarding stimuli in cocaine-dependent individuals. Twenty abstinent cocaine-dependent and 21 healthy matched male controls viewed erotic and neutral pictures while undergoing a functional magnetic resonance imaging scan. Group independent component analysis was then performed in order to investigate how functional networks were modulated by reward in cocaine addicts. The results showed that cocaine addicts, compared with healthy controls, displayed diminished modulation of the left frontoparietal network in response to erotic pictures, specifically when they were unpredicted. Additionally, a positive correlation between the length of cocaine abstinence and the modulation of the left frontoparietal network by unpredicted erotic images was found. In agreement with current addiction models, our results suggest that cocaine addiction contributes to reduce sensitivity to rewarding stimuli and that abstinence may mitigate this effect.

Dissociable Effects of Cocaine Dependence on Reward Processes: The Role of Acute Cocaine and Craving

The relative impact of chronic vs acute cocaine on dependence-related variability in reward processing in cocaine-dependent individuals (CD) is not well understood, despite the relevance of such effects to long-term outcomes. To dissociate these effects, CD (N=15) and healthy controls (HC; N=15) underwent MRI two times while performing a monetary incentive delay task. Both scans were identical across subjects/groups, except that, in a single-blind, counterbalanced design, CD received intravenous cocaine (30 mg/70 kg) before one session (CD+cocaine) and saline in another (CD+saline). Imaging analyses focused on activity related to anticipatory valence (gain/loss), anticipatory magnitude (small/medium/large), and reinforcing outcomes (successful/unsuccessful). Drug condition (cocaine vs saline) and group (HC vs CD+cocaine or CD+saline) did not influence valence-related activity. However, compared with HC, magnitude-related activity for gains was reduced in CD in the left cingulate gyrus post-cocaine and in the left putamen in the abstinence/saline condition. In contrast, magnitude-dependent activity for losses increased in CD vs HC in the right inferior parietal lobe post-cocaine and in the left superior frontal gyrus post-saline. Across outcomes (ie, successful and unsuccessful) activity in the right habenula decreased in CD in the abstinence/saline condition vs acute cocaine and HC. Cocaine-dependent variability in outcome- and loss-magnitude activity correlated negatively with ratings of cocaine craving and positively with how high subjects felt during the scanning session. Collectively, these data suggest dissociable effects of acute cocaine on non-drug reward processes, with cocaine consumption partially ameliorating dependence-related insensitivity to reinforcing outcomes/'liking', but having no discernible effect on dependence-related alterations in incentive salience/'wanting'. The relationship of drug-related affective sequelae to non-drug reward processing suggests that CD experience a general alteration of reward function and may be motivated to continue using cocaine for reasons beyond desired drug-related effects. This may have implications for individual differences in treatment efficacy for approaches that rely on reinforcement strategies (eg, contingency management) and for the long-term success of treatment.

Translational control of nicotine-evoked synaptic potentiation in mice and neuronal responses in human smokers by eIF2α

Adolescents are particularly vulnerable to nicotine, the principal addictive component driving tobacco smoking. In a companion study, we found that reduced activity of the translation initiation factor eIF2α underlies the hypersensitivity of adolescent mice to the effects of cocaine. Here we report that nicotine potentiates excitatory synaptic transmission in ventral tegmental area dopaminergic neurons more readily in adolescent mice compared to adults. Adult mice with genetic or pharmacological reduction in p-eIF2α-mediated translation are more susceptible to nicotine's synaptic effects, like adolescents. When we investigated the influence of allelic variability of the Eif2s1 gene (encoding eIF2α) on reward-related neuronal responses in human smokers, we found that a single nucleotide polymorphism in the Eif2s1 gene modulates mesolimbic neuronal reward responses in human smokers. These findings suggest that p-eIF2α regulates synaptic actions of nicotine in both mice and humans, and that reduced p-eIF2α may enhance susceptibility to nicotine (and other drugs of abuse) during adolescence.

Converging effects of cocaine addiction and sex on neural responses to monetary rewards

There is some evidence that cocaine addiction manifests as more severe in women than men. Here, we examined whether these sex-specific differences in the clinical setting parallel differential neurobehavioral sensitivity to rewards in the laboratory setting. Twenty-eight (14 females/14 males) cocaine-dependent and 25 (11 females/14 males) healthy individuals completed a monetary reward task during fMRI. Results showed that the effects of cocaine dependence and sex overlapped in regions traditionally considered part of the mesocorticolimbic brain circuits including the hippocampus and posterior cingulate cortex (PCC), as well as those outside of this circuit (e.g., the middle temporal gyrus). The nature of this 'overlap' was such that both illness and female sex were associated with lower activations in these regions in response to money. Diagnosis-by-sex interactions instead emerged in the frontal cortex, such that cocaine-dependent females exhibited lower precentral gyrus and greater inferior frontal gyrus (IFG) activations relative to cocaine-dependent males and healthy females. Within these regions modulated both by diagnosis and sex, lower activation in the hippocampus and PCC, and higher IFG activations, correlated with increased subjective craving during the task. Results suggest sex-specific differences in addiction extend to monetary rewards and may contribute to core symptoms linked to relapse.

Impaired neural response to negative prediction errors in cocaine addiction

Learning can be guided by unexpected success or failure, signaled via dopaminergic positive reward prediction error (+RPE) and negative reward-prediction error (-RPE) signals, respectively. Despite conflicting empirical evidence, RPE signaling is thought to be impaired in drug addiction. To resolve this outstanding question, we studied as a measure of RPE the feedback negativity (FN) that is sensitive to both reward and the violation of expectation. We examined FN in 25 healthy controls; 25 individuals with cocaine-use disorder (CUD) who tested positive for cocaine on the study day (CUD+), indicating cocaine use within the past 72 h; and in 25 individuals with CUD who tested negative for cocaine (CUD-). EEG was acquired while the participants performed a gambling task predicting whether they would win or lose money on each trial given three known win probabilities (25, 50, or 75%). FN was scored for the period in each trial when the actual outcome (win or loss) was revealed. A significant interaction between prediction, outcome, and group revealed that controls showed increased FN to unpredicted compared with predicted wins (i.e., intact +RPE) and decreased FN to unpredicted compared with predicted losses (i.e., intact -RPE). However, neither CUD subgroup showed FN modulation to loss (i.e., impaired -RPE), and unlike CUD+ individuals, CUD- individuals also did not show FN modulation to win (i.e., impaired +RPE). Thus, using FN, the current study directly documents -RPE deficits in CUD individuals. The mechanisms underlying -RPE signaling impairments in addiction may contribute to the disadvantageous nature of excessive drug use, which can persist despite repeated unfavorable life experiences (e.g., frequent incarcerations).

Disordered gambling: the evolving concept of behavioral addiction

The reclassification of gambling disorder within the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) addictions category marks an important step for addiction science. The similarities between gambling disorder and the substance use disorders have been well documented. As gambling is unlikely to exert actively damaging effects on the brain, the cognitive sequelae of gambling disorder may provide insights into addictive vulnerabilities; this idea is critically evaluated in light of recent structural imaging data. The second part of the review analyzes a fundamental question of how a behavior can become addictive in the absence of exogenous drug stimulation. The relative potency of drug and nondrug rewards is considered, alongside evidence that cognitive distortions in the processing of chance (for example, the illusion of control and the gambler's fallacy) may constitute an important added ingredient in gambling. Further understanding of these mechanisms at neural and behavioral levels will be critical for the classification of future behavioral addictions, and I consider the current research data for obesity and binge eating, compulsive shopping, and internet gaming disorder.