Learning that a cocaine reward is smaller than expected: A test of Redish's computational model of addiction

Generalized cue reactivity in dopamine neurons after opioids

Cue reactivity is the maladaptive neurobiological and behavioral response upon exposure to drug cues and is a major driver of relapse. The leading hypothesis is that dopamine release by addictive drugs represents a persistently positive reward prediction error that causes runaway enhancement of dopamine responses to drug cues, leading to their pathological overvaluation compared to non-drug reward alternatives. However, this hypothesis has not been directly tested. Here we developed Pavlovian and operant procedures to measure firing responses, within the same dopamine neurons, to drug versus natural reward cues, which we found to be similarly enhanced compared to cues predicting natural rewards in drug-naïve controls. This enhancement was associated with increased behavioral reactivity to the drug cue, suggesting that dopamine release is still critical to cue reactivity, albeit not as previously hypothesized. These results challenge the prevailing hypothesis of cue reactivity, warranting new models of dopaminergic function in drug addiction, and provide critical insights into the neurobiology of cue reactivity with potential implications for relapse prevention.

Computational models of drug use and addiction: A review

In this brief review, we describe current computational models of drug-use and addiction that fall into 2 broad categories: mathematically based models that rely on computational theories, and brain-based models that link computations to brain areas or circuits. Across categories, many are models of learning and decision-making, which may be compromised in addiction. Several mathematical models take predictive coding approaches, focusing on Bayesian prediction error. Other models focus on learning processes and (traditional) prediction error. Brain-based models have incorporated prefrontal cortex, basal ganglia, and the dopamine system, based on the effects of drugs on dopamine, motivation, and executive control circuits. Several models specifically describe how behavioral control may transition from habitual to goal-directed systems, consistent with computational accounts of compromised "model-based" control. Some brain-based models have linked this to the transition of behavioral control from ventral to dorsal striatum. Overall, we propose that while computational models capture some aspects of addiction and have advanced our thinking, most have focused on the effects of drug use rather than addiction per se, most have not been tested on and/or supported by human data, and few capture multiple stages and symptoms of addiction. We conclude by suggesting a path forward for computational models of addiction. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Beyond simple tests of value: measuring addiction as a heterogeneous disease of computation-specific valuation processes

Addiction is considered to be a neurobiological disorder of learning and memory because addiction is capable of producing lasting changes in the brain. Recovering addicts chronically struggle with making poor decisions that ultimately lead to relapse, suggesting a view of addiction also as a neurobiological disorder of decision-making information processing. How the brain makes decisions depends on how decision-making processes access information stored as memories in the brain. Advancements in circuit-dissection tools and recent theories in neuroeconomics suggest that neurally dissociable valuation processes access distinct memories differently, and thus are uniquely susceptible as the brain changes during addiction. If addiction is to be considered a neurobiological disorder of memory, and thus decision-making, the heterogeneity with which information is both stored and processed must be taken into account in addiction studies. Addiction etiology can vary widely from person to person. We propose that addiction is not a single disease, nor simply a disorder of learning and memory, but rather a collection of symptoms of heterogeneous neurobiological diseases of distinct circuit-computation-specific decision-making processes.

Reward Prediction Errors in Drug Addiction and Parkinson's Disease: from Neurophysiology to Neuroimaging

PURPOSE OF REVIEW

Surprises are important sources of learning. Cognitive scientists often refer to surprises as "reward prediction errors," a parameter that captures discrepancies between expectations and actual outcomes. Here, we integrate neurophysiological and functional magnetic resonance imaging (fMRI) results addressing the processing of reward prediction errors and how they might be altered in drug addiction and Parkinson's disease.

RECENT FINDINGS

By increasing phasic dopamine responses, drugs might accentuate prediction error signals, causing increases in fMRI activity in mesolimbic areas in response to drugs. Chronic substance dependence, by contrast, has been linked with compromised dopaminergic function, which might be associated with blunted fMRI responses to pleasant non-drug stimuli in mesocorticolimbic areas. In Parkinson's disease, dopamine replacement therapies seem to induce impairments in learning from negative outcomes. The present review provides a holistic overview of reward prediction errors across different pathologies and might inform future clinical strategies targeting impulsive/compulsive disorders.

Using model systems to understand errant plasticity mechanisms in psychiatric disorders

In vivo model systems are a critical tool for gaining insight into the pathology underlying psychiatric disorders. Although modern functional imaging tools allow study of brain correlates of behavior in clinical groups and genome-wide association studies are beginning to uncover the complex genetic architecture of psychiatric disorders, there is less understanding of pathology at intervening levels of organization. Several psychiatric disorders derive from pathological neural plasticity, and studying the mechanisms that underlie these processes, including reinforcement learning and spike-timing-dependent plasticity, requires the use of animals. It will be particularly important to understand how individual differences in plasticity mechanisms at a cellular level confer resilience on some but lead to disease in others.

Cocaine can generate a stronger conditioned reinforcer than food despite being a weaker primary reinforcer

The present study aimed to test the hypothesis that cues associated with drug-taking behavior become extra strong motivators of behavior compared with cues paired with non-drug reinforcers. In experiment 1, rats were trained to lever press for intravenous cocaine infusions and grain pellets. Each reinforcer was paired with a distinct audiovisual cue. When allowed to choose between these alternatives, rats chose grain on ~70-80 percent of trials. However, after extinguishing lever pressing, reintroduction of press-contingent cues during a test for cue-induced reinstatement generated more cocaine seeking than grain seeking (also observed on 3- and 8-week follow-up tests). To examine whether the same pattern of results would occur with two non-drug reinforcers, experiment 2 replicated experiment 1 using grain and sucrose as reinforcement alternatives. Rats chose sucrose over grain on ~70-80 percent of choice trials and also responded more for the sucrose cue than for the grain cue on the reinstatement test. The disconnect between primary and conditioned reinforcements in experiment 1 but not in experiment 2 suggests that drug cues may become exceptionally strong motivators of drug seeking. These results are consistent with cue-focused theories of addiction and may offer insight into the persistent cue-driven drug-seeking behavior observed in addiction.

Decision making: from neuroscience to psychiatry

Adaptive behaviors increase the likelihood of survival and reproduction and improve the quality of life. However, it is often difficult to identify optimal behaviors in real life due to the complexity of the decision maker's environment and social dynamics. As a result, although many different brain areas and circuits are involved in decision making, evolutionary and learning solutions adopted by individual decision makers sometimes produce suboptimal outcomes. Although these problems are exacerbated in numerous neurological and psychiatric disorders, their underlying neurobiological causes remain incompletely understood. In this review, theoretical frameworks in economics and machine learning and their applications in recent behavioral and neurobiological studies are summarized. Examples of such applications in clinical domains are also discussed for substance abuse, Parkinson's disease, attention-deficit/hyperactivity disorder, schizophrenia, mood disorders, and autism. Findings from these studies have begun to lay the foundations necessary to improve diagnostics and treatment for various neurological and psychiatric disorders.

Assessment of the impact of pattern of cocaine dosing schedule during conditioning and reconditioning on magnitude of cocaine CPP, extinction, and reinstatement

RATIONALE AND OBJECTIVE

We sought to examine the impact of differing cocaine administration schedules and dosing on the magnitude of cocaine conditioned place preference (CPP), extinction, and stress- and cocaine-induced reinstatement of CPP.

METHODS

First, in C57Bl/6J mice, we investigated whether total cocaine administration or pattern of drug exposure could influence the magnitude of cocaine CPP by conditioning mice with a fixed-low dose (FL; 7.5 mg/kg; total of 30 mg/kg), a fixed-high dose (FH; 16 mg/kg; total of 64 mg/kg), or an ascending dosing schedule (Asc; 2, 4, 8, and 16 mg/kg; total of 30 mg/kg). Next, we investigated if cocaine or saline is more effective at extinguishing preference by reconditioning mice with either a descending dosing schedule (Desc; 8, 4, 2, and 1 mg/kg) or saline. Finally, we examined if prior conditioning and reconditioning history alters stress (~2-3-min forced swim test) or cocaine-induced (3.5 mg/kg) reinstatement.

RESULTS

We replicated and extended findings by Itzhak and Anderson (Addict. Biol. 17(4): 706-16, 2011) demonstrating that Asc conditioning produces a greater CPP than either the FL or FH conditioning schedules. The magnitude of extinction expressed was similar in the Desc reconditioned and saline groups. Moreover, only the saline, and not the Desc reconditioned mice, showed stress and cocaine-induced reinstatement of CPP.

CONCLUSIONS

Our results suggest that the schedule of cocaine administration during conditioning and reconditioning can have a significant influence on the magnitude of CPP and extinction of preference and the ability of cocaine or a stressor to reinstate CPP.

A Java simulator of Rescorla and Wagner's prediction error model and configural cue extensions

In this paper we present the "R&W Simulator" (version 3.0), a Java simulator of Rescorla and Wagner's prediction error model of learning. It is able to run whole experimental designs, and compute and display the associative values of elemental and compound stimuli simultaneously, as well as use extra configural cues in generating compound values; it also permits change of the US parameters across phases. The simulator produces both numerical and graphical outputs, and includes a functionality to export the results to a data processor spreadsheet. It is user-friendly, and built with a graphical interface designed to allow neuroscience researchers to input the data in their own "language". It is a cross-platform simulator, so it does not require any special equipment, operative system or support program, and does not need installation. The "R&W Simulator" (version 3.0) is available free.

Changes in the magnitude of drug-unconditioned stimulus during conditioning modulate cocaine-induced place preference in mice

Drug reinforcement learning is relevant for the development of addiction. The present study investigated how changes in the magnitude of drug-unconditioned stimulus during associative learning modulate the acquisition and extinction of cocaine-induced conditioned place preference (CPP). B6;129S F2 mice were conditioned by three dosing schedules of cocaine: (1) ascending, (2) fixed and (3) descending daily doses. Following acquisition of CPP, extinction was induced by (1) context re-exposure, (2) reconditioning by saline and (3) reconditioning by descending doses of cocaine. The magnitude of CPP following conditioning by daily ascending doses of cocaine (2, 4, 8 and 16 mg/kg) was significantly higher than that obtained from conditioning by either a fixed daily dose (16 mg/kg × 4 days) or daily descending doses (24, 12, 6 and 3 mg/kg). Extinction following context re-exposure showed persistent CPP in the 'ascending' group compared to the other two groups. However, extinction via reconditioning by saline was equally effective in all groups. Interestingly, reconditioning by descending doses of cocaine (1) extinguished CPP and (2) resulted in partial resistance to the reinstatement of conditioned response by cocaine priming. Results underscore the significance of daily changes in cocaine dosage in the development and extinction of drug-induced conditioned response. Increase and decrease in cocaine dosage strengthens and weakens cocaine-associated memory, respectively. Moreover, extinction by 'tapering down' drug reward may be superior to extinction by saline.

Drug intake is sufficient, but conditioning is not necessary for the emergence of compulsive cocaine seeking after extended self-administration

Compulsive drug seeking, which is characterized by continued instrumental effort despite contingent punishment, has been shown to emerge after extended drug self-administration. Exactly what aspect of drug self-administration drives the appearance of addictive behavior is unclear, but the mechanistic explanations that have been offered differ in one key respect. On one hand, it has been suggested that dysfunctional conditioning during self-administration drives unrealistic reward expectations, ultimately producing resistance to punishment. If this is indeed the pathological process that drives compulsive behavior, then compulsivity should be apparent only in the presence of the pavlovian and instrumental stimuli that underwent frequent pairing with the drug reward. On the other hand, it has also been suggested that extended drug intake produces general changes to reward and decision-making circuits that manifest as compulsive drug seeking. Unfortunately, conditioning history and drug intake are generally intrinsically intertwined. However, here we used an animal model of compulsive cocaine seeking to selectively manipulate drug intake and the degree of conditioning in the test context, to investigate which of the two is more important for the emergence of compulsive cocaine seeking. The results show that extended drug intake alone is sufficient, but extended conditioning in the test context is not necessary for the emergence of compulsive cocaine seeking, resolving a fundamental question in addiction research.