Impairments in reinforcement learning do not explain enhanced habit formation in cocaine use disorder

Medial prefrontal cortex suppresses reward-seeking behavior with risk of punishment by reducing sensitivity to reward

Reward-seeking behavior is frequently associated with risk of punishment. There are two types of punishment: positive punishment, which is defined as addition of an aversive stimulus, and negative punishment, involves the omission of a rewarding outcome. Although the medial prefrontal cortex (mPFC) is important in avoiding punishment, whether it is important for avoiding both positive and negative punishment and how it contributes to such avoidance are not clear. In this study, we trained male mice to perform decision-making tasks under the risks of positive (air-puff stimulus) and negative (reward omission) punishment, and modeled their behavior with reinforcement learning. Following the training, we pharmacologically inhibited the mPFC. We found that pharmacological inactivation of mPFC enhanced the reward-seeking choice under the risk of positive, but not negative, punishment. In reinforcement learning models, this behavioral change was well-explained as an increase in sensitivity to reward, rather than a decrease in the strength of aversion to punishment. Our results suggest that mPFC suppresses reward-seeking behavior by reducing sensitivity to reward under the risk of positive punishment.

Listening to the Data: Computational Approaches to Addiction and Learning

Computational approaches hold great promise for identifying novel treatment targets and creating translational therapeutics for substance use disorders. From circuitries underlying decision-making to computationally derived neural markers of drug-cue reactivity, this review is a summary of the approaches to data presented at our 2023 Society for Neuroscience Mini-Symposium. Here, we highlight data- and hypothesis-driven computational approaches that recently afforded advancements in addiction and learning neuroscience. First, we discuss the value of hypothesis-driven algorithmic modeling approaches, which integrate behavioral, neural, and cognitive outputs to refine hypothesis testing. Then, we review the advantages of data-driven dimensionality reduction and machine learning methods for uncovering novel predictor variables and elucidating relationships in high-dimensional data. Overall, this review highlights recent breakthroughs in cognitive mapping, model-based analysis of behavior/risky decision-making, patterns of drug taking, relapse, and neuromarker discovery, and showcases the benefits of novel modeling techniques, across both preclinical and clinical data.

Methylphenidate as a treatment option for substance use disorder: a transdiagnostic perspective

A transition in viewing mental disorders from conditions defined as a set of unique characteristics to one of the quantitative variations on a collection of dimensions allows overlap between disorders. The overlap can be utilized to extend to treatment approaches. Here, we consider the overlap between attention-deficit/hyperactivity disorder and substance use disorder to probe the suitability to use methylphenidate as a treatment for substance use disorder. Both disorders are characterized by maladaptive goal-directed behavior, impaired cognitive control, hyperactive phasic dopaminergic neurotransmission in the striatum, prefrontal hypoactivation, and reduced frontal cortex gray matter volume/density. In addition, methylphenidate has been shown to improve cognitive control and normalize associated brain activation in substance use disorder patients and clinical trials have found methylphenidate to improve clinical outcomes. Despite the theoretical basis and promising, but preliminary, outcomes, many questions remain unanswered. Most prominent is whether all patients who are addicted to different substances may equally profit from methylphenidate treatment.

Challenging social media threats using collective well-being-aware recommendation algorithms and an educational virtual companion

Social media have become an integral part of our lives, expanding our interlinking capabilities to new levels. There is plenty to be said about their positive effects. On the other hand, however, some serious negative implications of social media have been repeatedly highlighted in recent years, pointing at various threats to society and its more vulnerable members, such as teenagers, in particular, ranging from much-discussed problems such as digital addiction and polarization to manipulative influences of algorithms and further to more teenager-specific issues (e.g., body stereotyping). The impact of social media-both at an individual and societal level-is characterized by the complex interplay between the users' interactions and the intelligent components of the platform. Thus, users' understanding of social media mechanisms plays a determinant role. We thus propose a theoretical framework based on an adaptive "Social Media Virtual Companion" for educating and supporting an entire community, teenage students, to interact in social media environments in order to achieve desirable conditions, defined in terms of a community-specific and participatory designed measure of Collective Well-Being (CWB). This Companion combines automatic processing with expert intervention and guidance. The virtual Companion will be powered by a Recommender System (CWB-RS) that will optimize a CWB metric instead of engagement or platform profit, which currently largely drives recommender systems thereby disregarding any societal collateral effect. CWB-RS will optimize CWB both in the short term by balancing the level of social media threats the users are exposed to, and in the long term by adopting an Intelligent Tutor System role and enabling adaptive and personalized sequencing of playful learning activities. We put an emphasis on experts and educators in the educationally managed social media community of the Companion. They play five key roles: (a) use the Companion in classroom-based educational activities; (b) guide the definition of the CWB; (c) provide a hierarchical structure of learning strategies, objectives and activities that will support and contain the adaptive sequencing algorithms of the CWB-RS based on hierarchical reinforcement learning; (d) act as moderators of direct conflicts between the members of the community; and, finally, (e) monitor and address ethical and educational issues that are beyond the intelligent agent's competence and control. This framework offers a possible approach to understanding how to design social media systems and embedded educational interventions that favor a more healthy and positive society. Preliminary results on the performance of the Companion's components and studies of the educational and psychological underlying principles are presented.

Impaired belief updating and devaluation in adult women with bulimia nervosa

Recent models of bulimia nervosa (BN) propose that binge-purge episodes ultimately become automatic in response to cues and insensitive to negative outcomes. Here, we examined whether women with BN show alterations in instrumental learning and devaluation sensitivity using traditional and computational modeling analyses of behavioral data. Adult women with BN (n = 30) and group-matched healthy controls (n = 31) completed a task in which they first learned stimulus-response-outcome associations. Then, participants were required to repeatedly adjust their responses in a "baseline test", when different sets of stimuli were explicitly devalued, and in a "slips-of-action test", when outcomes instead of stimuli were devalued. The BN group showed intact behavioral sensitivity to outcome devaluation during the slips-of-action test, but showed difficulty overriding previously learned stimulus-response associations on the baseline test. Results from a Bayesian learner model indicated that this impaired performance could be accounted for by a slower pace of belief updating when a new set of previously learned responses had to be inhibited (p = 0.036). Worse performance and a slower belief update in the baseline test were each associated with more frequent binge eating (p = 0.012) and purging (p = 0.002). Our findings suggest that BN diagnosis and severity are associated with deficits in flexibly updating beliefs to withhold previously learned responses to cues. Additional research is needed to determine whether this impaired ability to adjust behavior is responsible for maintaining automatic and persistent binge eating and purging in response to internal and environmental cues.

The effects of acute oral naltrexone pretreatment on the abuse potential of intranasal methamphetamine, and the relationship between reward/punishment sensitivity and methamphetamine's effects

One potential medication for treating methamphetamine use disorder is the opioid antagonist naltrexone (NLTX). Despite encouraging preclinical findings, the results of clinical studies have been mixed. The primary aim of the current trial was to examine the effects of acute NLTX pretreatment on the subjective and reinforcing effects of intranasal methamphetamine. Nonmedical psychostimulant users completed outpatient testing sessions in which they received oral placebo (0 mg) or NLTX (50 mg) before intranasal methamphetamine (30 mg/70 kg). Primary outcome measures were peak positive subjective effects (e.g. drug 'Liking') assessed on a visual analog scale (0-100), and methamphetamine self-administration using an operant self-administration task. Participants also completed a probabilistic categorization task to assess reward and punishment learning sensitivity. Complete data were available from 13 male and 1 transgender (male-to-female) participant (age: 33.4 ± 7.6 years). Intranasal methamphetamine significantly increased subjective ratings of drug 'Liking', 'Good Effect' and 'High' from baseline (P's < 0.01), but did not significantly vary as a function of placebo or NLTX pretreatment. Similarly, methamphetamine self-administration did not vary between the placebo and NLTX pretreatment conditions. This sample did not demonstrate a significant 'bias' in learning from positive and negative outcomes (i.e. reward and punishment sensitivity), and reward/punishment sensitivity was not correlated with the effects of methamphetamine or the effects of NLTX on methamphetamine. The current study argues against the use of NLTX as a stand-alone medication for treating methamphetamine use disorder.

Impaired Learning From Negative Feedback in Stimulant Use Disorder: Dopaminergic Modulation

BACKGROUND

Drug-induced alterations to the dopamine system in stimulant use disorder (SUD) are hypothesized to impair reinforcement learning (RL). Computational modeling enables the investigation of the latent processes of RL in SUD patients, which could elucidate the nature of their impairments.

METHODS

We investigated RL in 44 SUD patients and 41 healthy control participants using a probabilistic RL task that assesses learning from reward and punishment separately. In an independent sample, we determined the modulatory role of dopamine in RL following a single dose of the dopamine D2/3 receptor antagonist amisulpride (400 mg) and the agonist pramipexole (0.5 mg) in a randomised, double-blind, placebo-controlled, crossover design. We analyzed task performance using computational modelling and hypothesized that RL impairments in SUD patients would be differentially modulated by a dopamine D2/3 receptor antagonist and agonist.

RESULTS

Computational analyses in both samples revealed significantly reduced learning rates from punishment in SUD patients compared with healthy controls, whilst their reward learning rates were not measurably impaired. In addition, the dopaminergic receptor agents modulated RL parameters differentially in both groups. Both amisulpride and pramipexole impaired RL parameters in healthy participants, but ameliorated learning from punishment in SUD patients.

CONCLUSION

Our findings suggest that RL impairments seen in SUD patients are associated with altered dopamine function.

Sex Differences and Exogenous Estrogen Influence Learning and Brain Responses to Prediction Errors

Animal studies show marked sex differences as well as effects of estrogen (E2) in the mesocorticolimbic dopaminergic (DA) pathways, which play a critical role in reward processing and reinforcement learning and are also implicated in drug addiction. In this computational pharmacological fMRI study, we investigate the effects of both factors, sex and estrogen, on reinforcement learning and the dopaminergic system in humans; 67 male and 64 naturally cycling female volunteers, the latter in their low-hormone phase, were randomly assigned, double-blind, to take E2 or placebo. They completed a reinforcement learning task in the MRI scanner for which we have previously shown reward prediction error (RPE)-related activity to be dopaminergic. We found RPE-related brain activity to be enhanced in women compared with men and to a greater extent when E2 levels were elevated in both sexes. However, both factors, female sex and E2, slowed adaptation to RPEs (smaller learning rate). This discrepancy of larger RPE-related activity yet smaller learning rates can be explained by organizational sex differences and activational effects of circulating E2, which both affect DA release differently to DA receptor binding capacities.

Cell Adhesion Factors in the Orbitofrontal Cortex Control Cue-Induced Reinstatement of Cocaine Seeking and Amygdala-Dependent Goal Seeking

Repeated cocaine exposure causes dendritic spine loss in the orbitofrontal cortex, which might contribute to poor orbitofrontal cortical function following drug exposure. One challenge, however, has been verifying links between neuronal structural plasticity and behavior, if any. Here we report that cocaine self-administration triggers the loss of dendritic spines on excitatory neurons in the orbitofrontal cortex of male and female mice (as has been reported in rats). To understand functional consequences, we locally ablated neuronal β1-integrins, cell adhesion receptors that adhere cells to the extracellular matrix and thus support dendritic spine stability. Degradation of β1-integrin tone: (1) caused dendritic spine loss, (2) exaggerated cocaine-seeking responses in a cue-induced reinstatement test, and (3) impaired the ability of mice to integrate new learning into familiar routines, a key function of the orbitofrontal cortex. Stimulating Abl-related gene kinase, overexpressing Proline-rich tyrosine kinase, and inhibiting Rho-associated coiled-coil containing kinase corrected response strategies, uncovering a β1-integrin-mediated signaling axis that controls orbitofrontal cortical function. Finally, use of a combinatorial gene silencing/chemogenetic strategy revealed that β1-integrins support the ability of mice to integrate new information into established behaviors by sustaining orbitofrontal cortical connections with the basolateral amygdala.SIGNIFICANCE STATEMENT Cocaine degenerates dendritic spines in the orbitofrontal cortex, a region of the brain involved in interlacing new information into established behaviors. One challenge has been verifying links between cellular structural stability and behavior, if any. In this second of two related investigations, we study integrin family receptors, which adhere cells to the extracellular matrix and thereby stabilize dendritic spines (see also DePoy et al., 2019). We reveal that β1-integrins in the orbitofrontal cortex control food- and cocaine-seeking behaviors. For instance, β1-integrin loss amplifies cocaine-seeking behavior and impairs the ability of mice to integrate new learning into familiar routines. We identify likely intracellular signaling partners by which β1-integrins support orbitofrontal cortical function and connectivity with the basolateral amygdala.

[The sensorimotor domain in the research domain criteria system: progress and perspectives]

Over the past three decades research interest in hypokinetic, hyperkinetic, sensorimotor and psychomotor abnormalities in mental disorders has steadily increased. This development has led to an increasing number of scientific initiatives that have not only highlighted the clinical need for early detection of extrapyramidal motor symptoms, tardive dyskinesia and catatonia but also provided numerous neurobiological findings and clinically relevant results based on the pathology of the sensorimotor system in patients with mental disorders. In view of these developments in January 2019 the National Institute of Mental Health (NIMH) research domain criteria (RDoC) initiative introduced a sixth domain called the sensorimotor domain to address deficits in the sensorimotor system and associated behavioral abnormalities. To draw attention to the rapid progress just since the introduction of the sensorimotor domain, a 2-year (1 January 2019-18 February 2021) systematic review is presented highlighting recent neuroimaging findings and discussing challenges for future research. In summary, aberrant sensorimotor processing in mental disorders is associated with dysfunction of the cerebello-thalamo-motor cortex network, which interacts with (social)cognitive and affective systems. Initial longitudinal and interventional studies highlight the translational potential of the sensorimotor domain.

Reduced Glutamate Turnover in the Putamen Is Linked With Automatic Habits in Human Cocaine Addiction

BACKGROUND

The balance between goal-directed behavior and habits has been hypothesized to be biased toward the latter in individuals with cocaine use disorder (CUD), suggesting possible neurochemical changes in the putamen, which may contribute to their compulsive behavior.

METHODS

We assessed habitual behavior in 48 patients with CUD and 42 healthy control participants using a contingency degradation paradigm and the Creature of Habit Scale. In a subgroup of this sample (CUD: n = 21; control participants: n = 22), we also measured glutamate and glutamine concentrations in the left putamen using ultra-high-field (7T) magnetic resonance spectroscopy. We hypothesized that increased habitual tendencies in patients with CUD would be associated with abnormal glutamatergic metabolites in the putamen.

RESULTS

Compared with their non-drug-using peers, patients with CUD exhibited greater habitual tendencies during contingency degradation, which correlated with increased levels of self-reported daily habits. We further identified a significant reduction in glutamate concentration and glutamate turnover (glutamate-to-glutamine ratio) in the putamen in patients with CUD, which was significantly related to the level of self-reported daily habits.

CONCLUSIONS

Patients with CUD exhibit enhanced habitual behavior, as assessed both by questionnaire and by a laboratory paradigm of contingency degradation. This automatic habitual tendency is related to a reduced glutamate turnover in the putamen, suggesting a dysregulation of habits caused by chronic cocaine use.

Single Exposure to Cocaine Impairs Reinforcement Learning by Potentiating the Activity of Neurons in the Direct Striatal Pathway in Mice

Plasticity in the glutamatergic synapses on striatal medium spiny neurons (MSNs) is not only essential for behavioral adaptation but also extremely vulnerable to drugs of abuse. Modulation on these synapses by even a single exposure to an addictive drug may interfere with the plasticity required by behavioral learning and thus produce impairment. In the present work, we found that the negative reinforcement learning, escaping mild foot-shocks by correct nose-poking, was impaired by a single in vivo exposure to 20 mg/kg cocaine 24 h before the learning in mice. Either a single exposure to cocaine or reinforcement learning potentiates the glutamatergic synapses on MSNs expressing the striatal dopamine 1 (D1) receptor (D1-MSNs). However, 24 h after the cocaine exposure, the potentiation required for reinforcement learning was disrupted. Specific manipulation of the activity of striatal D1-MSNs in D1-cre mice demonstrated that activation of these MSNs impaired reinforcement learning in normal D1-cre mice, but inhibition of these neurons reversed the reinforcement learning impairment induced by cocaine. The results suggest that cocaine potentiates the activity of direct pathway neurons in the dorsomedial striatum and this potentiation might disrupt the potentiation produced during and required for reinforcement learning.

Computational theory-driven studies of reinforcement learning and decision-making in addiction: What have we learned?

Computational psychiatry provides a powerful new approach for linking the behavioral manifestations of addiction to their precise cognitive and neurobiological substrates. However, this emerging area of research is still limited in important ways. While research has identified features of reinforcement learning and decision-making in substance users that differ from health, less emphasis has been placed on capturing addiction cycles/states dynamically, within-person. In addition, the focus on few behavioral variables at a time has precluded more detailed consideration of related processes and heterogeneous clinical profiles. We propose that a longitudinal and multidimensional examination of value-based processes, a type of dynamic "computational fingerprint", will provide a more complete understanding of addiction as well as aid in developing better tailored and timed interventions.

Astrocytic equilibrative nucleoside transporter type 1 upregulations in the dorsomedial and dorsolateral striatum distinctly coordinate goal-directed and habitual ethanol-seeking behaviours in mice

Two distinct dorsal striatum regions, dorsomedial striatum (DMS) and dorsolateral striatum (DLS), are attributed to conditioned goal-directed and habitual reward-seeking behaviours, respectively. Previously, our study shows that the ethanol-sensitive adenosine transporter, equilibrative nucleoside transporter 1 (ENT1), regulates ethanol-drinking behaviours. Although ENT1 is expressed in both neurons and astrocytes, astrocytic ENT1 is thought to regulate adenosine levels in response to ethanol. However, the role of DMS and DLS astrocytic ENT1 in goal-directed and habitual ethanol-seeking is not well known. Here, we identified whether the upregulation of astrocytic ENT1 in the DMS and DLS differentially regulates operant seeking behaviours towards the 10% sucrose (10S); 10% ethanol and 10% sucrose (10E10S); and 10% ethanol (10E) in mice. Using 4 days of random interval (RI), mice exhibited habitual seeking for 10S, but goal-directed seeking towards 10E10S. Using the same mice conditioned with 10E10S, we examined 10E-seeking behaviour on a fixed ratio (FR) for 6 days and RI for 8 days. On the other hand, during FR and the first 4 days of RI schedules, mice showed goal-directed seeking for 10E, whereas mice exhibited habitual seeking for 10E during the last 4 days of RI schedule. Interestingly, DMS astrocytic ENT1 upregulation promotes shift from habitual to goal-directed reward-seeking behaviours. By contrast, DLS astrocytic ENT1 upregulation showed no effects on behavioural shift. Taken together, our findings demonstrate that DMS astrocytic ENT1 contributes to reward-seeking behaviours.