Model-based and model-free decisions in alcohol dependence

On the effects of impulsivity and compulsivity on neural correlates of model-based performance

Impaired goal-directed behavior is associated with a range of mental disorders, implicating underlying transdiagnostic factors. While compulsivity has been linked to reduced model-based (MB) control, impulsivity has rarely been studied in the context of reinforcement learning despite its links to reward processing and cognitive control. This study investigated the neural mechanisms underlying MB control and the influence of impulsivity and compulsivity, using EEG data from 238 individuals during a two-step decision making task. Single-trial analyses revealed a modulation of the feedback-related negativity (FRN), where amplitudes were higher after common transitions and positive reward prediction error (RPE), indicating a valence effect. Meanwhile, enhanced P3 amplitudes after rare transitions and both positive and negative RPE possibly reflect surprise. In a second step, we regressed the mean b values of the effect of RPE on the EEG signals onto self-reported impulsivity and compulsivity and behavioral MB control (w). The effect of RPE on FRN-related activity was mainly associated with higher w scores, linking the FRN to MB control. Crucially, the modulation of the P3 by RPE was negatively associated with compulsivity, pointing to a deficient mental model in highly compulsive individuals.

Motivated reasoning and scientific racism in compulsion theory of human addiction: Methodological framework to promote social justice

Heinz et al. (2024) recently criticised habit/compulsion theory of human addiction but nevertheless concluded that 'habit formation plays a significant role in drug addiction'. To challenge this causal claim, the current article develops four further methodological criticisms, that publications supporting the habit/compulsion account of human addiction: (1) under-report contradictory observations; (2) exaggerate the process purity of positive observations; (3) under-emphasise the low quality of epidemiological support for a causal hypothesis; (4) recapitulate the social injustice of racial intelligence era by prematurely attributing lower task performance to drug user group membership (endophenotype) without having adequately tested social, psychological, economic and environmental inequalities. Methodological guidelines are recommended to address each concern, which should raise evidence standards, incorporate social justice and improve accuracy of estimating any specific effect of addiction history on task performance. Given that construing drug users as intellectually impaired could promote stigma and reduce their recovery potential, it is recommended that scientific discourse about habit/compulsive endophenotypes underpinning addiction is avoided until these higher evidence standards are met.

Disorders of compulsivity: Deficits in arbitrating learning strategies

While previous research has shown that compulsivity is related to an imbalance between goal-directed and habitual learning systems, very little is known about whether this effect is due to the impairment of a single system or the impairment of the arbitration mechanism that determines which system controls behaviour at any given moment; the current study aims to address this disagreement. Nineteen alcohol use disorder, 30 obsessive-compulsive disorder (OCD) and 20 major depressive disorder patients and corresponding sex- and age-matched controls performed two-choice, three-stage Markov decision-making paradigm. Model-based and mode-free reinforcement learning models were used to independently fitted their behavioural data. Alcohol use disorder and OCD patients showed less model-based strategy choice than healthy controls in task conditions where the model-based strategy was optimal. Only OCD patients showed higher behavioural control system switching in task conditions where model-free use was optimal. Major depressive disorder patients did not differ from the matched control in both. These findings suggest that dysfunction in arbitration control between dual systems may be the basis for diverse disorders involving compulsivity.

Bad habits-good goals? Meta-analysis and translation of the habit construct to alcoholism

Excessive alcohol consumption remains a global public health crisis, with millions suffering from alcohol use disorder (AUD, or simply "alcoholism"), leading to significantly reduced life expectancy. This review examines the interplay between habitual and goal-directed behaviors and the associated neurobiological changes induced by chronic alcohol exposure. Contrary to a strict habit-goal dichotomy, our meta-analysis of the published animal experiments combined with a review of human studies reveals a nuanced transition between these behavioral control systems, emphasizing the need for refined terminology to capture the probabilistic nature of decision biases in individuals with a history of chronic alcohol exposure. Furthermore, we distinguish habitual responding from compulsivity, viewing them as separate entities with diverse roles throughout the stages of the addiction cycle. By addressing species-specific differences and translational challenges in habit research, we provide insights to enhance future investigations and inform strategies for combatting AUD.

Lasting dynamic effects of the psychedelic 2,5-dimethoxy-4-iodoamphetamine ((±)-DOI) on cognitive flexibility

Psychedelic drugs can aid fast and lasting remission from various neuropsychiatric disorders, though the underlying mechanisms remain unclear. Preclinical studies suggest serotonergic psychedelics enhance neuronal plasticity, but whether neuroplastic changes can also be seen at cognitive and behavioural levels is unexplored. Here we show that a single dose of the psychedelic 2,5-dimethoxy-4-iodoamphetamine ((±)-DOI) affects structural brain plasticity and cognitive flexibility in young adult mice beyond the acute drug experience. Using ex vivo magnetic resonance imaging, we show increased volumes of several sensory and association areas one day after systemic administration of 2 mgkg-1 (±)-DOI. We then demonstrate lasting effects of (±)-DOI on cognitive flexibility in a two-step probabilistic reversal learning task where 2 mgkg-1 (±)-DOI improved the rate of adaptation to a novel reversal in task structure occurring one-week post-treatment. Strikingly, (±)-DOI-treated mice started learning from reward omissions, a unique strategy not typically seen in mice in this task, suggesting heightened sensitivity to previously overlooked cues. Crucially, further experiments revealed that (±)-DOI's effects on cognitive flexibility were contingent on the timing between drug treatment and the novel reversal, as well as on the nature of the intervening experience. (±)-DOI's facilitation of both cognitive adaptation and novel thinking strategies may contribute to the clinical benefits of psychedelic-assisted therapy, particularly in cases of perseverative behaviours and a resistance to change seen in depression, anxiety, or addiction. Furthermore, our findings highlight the crucial role of time-dependent neuroplasticity and the influence of experiential factors in shaping the therapeutic potential of psychedelic interventions for impaired cognitive flexibility.

Does compulsion explain addiction?

One of the leading drug addiction theories states that habits and the underlying neural process of a ventral to dorsal striatal shift are the building blocks of compulsive drug-seeking behaviour and that compulsion is the maladaptive persistence of responding despite adverse consequences. Here we discuss that compulsive behaviour as defined primarily from the perspective of animal experimentation falls short of the clinical phenomena and their neurobiological correlates. Thus for the human condition, the concept of compulsive habbits should be critically addressed and potentially revised.

Chronic Ethanol Exposure Produces Persistent Impairment in Cognitive Flexibility and Decision Signals in the Striatum

Lack of cognitive flexibility is a hallmark of substance use disorders and has been associated with drug-induced synaptic plasticity in the dorsomedial striatum (DMS). Yet the possible impact of altered plasticity on real-time striatal neural dynamics during decision-making is unclear. Here, we identified persistent impairments induced by chronic ethanol (EtOH) exposure on cognitive flexibility and striatal decision signals. After a substantial withdrawal period from prior EtOH vapor exposure, male, but not female, rats exhibited reduced adaptability and exploratory behavior during a dynamic decision-making task. Reinforcement learning models showed that prior EtOH exposure enhanced learning from rewards over omissions. Notably, neural signals in the DMS related to the decision outcome were enhanced, while those related to choice and choice-outcome conjunction were reduced, in EtOH-treated rats compared to the controls. These findings highlight the profound impact of chronic EtOH exposure on adaptive decision-making, pinpointing specific changes in striatal representations of actions and outcomes as underlying mechanisms for cognitive deficits.

The utility of a latent-cause framework for understanding addiction phenomena

Computational models of addiction often rely on a model-free reinforcement learning (RL) formulation, owing to the close associations between model-free RL, habitual behavior and the dopaminergic system. However, such formulations typically do not capture key recurrent features of addiction phenomena such as craving and relapse. Moreover, they cannot account for goal-directed aspects of addiction that necessitate contrasting, model-based formulations. Here we synthesize a growing body of evidence and propose that a latent-cause framework can help unify our understanding of several recurrent phenomena in addiction, by viewing them as the inferred return of previous, persistent "latent causes". We demonstrate that applying this framework to Pavlovian and instrumental settings can help account for defining features of craving and relapse such as outcome-specificity, generalization, and cyclical dynamics. Finally, we argue that this framework can bridge model-free and model-based formulations, and account for individual variability in phenomenology by accommodating the memories, beliefs, and goals of those living with addiction, motivating a centering of the individual, subjective experience of addiction and recovery.

Model-based and model-free mechanisms in methamphetamine use disorder

People with methamphetamine use disorder (MUD) struggle to shift their behaviour from methamphetamine-orientated habits to goal-oriented choices. The model-based/model-free framework is well suited to understand this difficulty by unpacking the computational mechanisms that support experienced-based (model-free) and goal-directed (model-based) choices. We aimed to examine whether 1) participants with MUD differed from controls on behavioural proxies and/or computational mechanisms of model-based/model-free choices; 2) model-based/model-free decision-making correlated with MUD symptoms; and 3) model-based/model-free deficits improved over six weeks in the group with MUD. Participants with MUD and controls with similar age, IQ and socioeconomic status completed the Two-Step Task at treatment commencement (MUD n = 30, Controls n = 31) and six weeks later (MUD n = 23, Controls n = 26). We examined behavioural proxies of model-based/model-free decisions using mixed logistic regression, and their underlying mechanisms using computational modelling. At a behavioural level, participants with MUD were more likely to switch their choices following rewarded actions, although this pattern improved at follow up. At a computational level, groups were similar in their use of model-based mechanisms, but participants with MUD were less likely to apply model-free mechanisms and less likely to repeat rewarded actions. We did not find evidence that individual differences in model-based or model-free parameters were associated with greater severity of methamphetamine dependence, nor did we find that group differences in computational parameters changed between baseline and follow-up assessment. Decision-making challenges in people with MUD are likely related to difficulties in pursuing choices previously associated with positive outcomes.

A feasibility study of goal-directed network-based real-time fMRI neurofeedback for anhedonic depression

Anhedonia is a hallmark symptom of depression that often lacks adequate interventions. The translational gap remains in clinical treatments based on neural substrates of anhedonia. Our pilot study found that depressed individuals depended less on goal-directed (GD) reward learning (RL), with reduced reward prediction error (RPE) BOLD signal. Previous studies have found that anhedonia is related to abnormal activities and/or functional connectivities of the central executive network (CEN) and salience network (SN), both of which belong to the goal-directed system. In addition, it was found that real-time functional magnetic resonance imaging (rt-fMRI) neurofeedback (NF) could improve the balance between CEN and SN in healthy individuals. Therefore, we speculate that rt-fMRI NF of the CEN and SN associated with the GD system may improve depressive and/or anhedonic symptoms. Therefore, this study (1) will examine individuals with anhedonic depression using GD-RL behavioral task, combined with functional magnetic resonance imaging and computational modeling to explore the role of CEN/SN deficits in anhedonic depression; and (2) will utilize network-based rt-fMRI NF to investigate whether it is feasible to regulate the differential signals of brain CEN/SN of GD system through rt-fMRI NF to alleviate depressive and/or anhedonic symptoms. This study highlights the need to elucidate the intervention effects of rt-fMRI NF and the underlying computational network neural mechanisms.

Shorter planning depth and higher response noise during sequential decision-making in old age

Forward planning is crucial to maximize outcome in complex sequential decision-making scenarios. In this cross-sectional study, we were particularly interested in age-related differences of forward planning. We presumed that especially older individuals would show a shorter planning depth to keep the costs of model-based decision-making within limits. To test this hypothesis, we developed a sequential decision-making task to assess forward planning in younger (age < 40 years; n = 25) and older (age > 60 years; n = 27) adults. By using reinforcement learning modelling, we inferred planning depths from participants' choices. Our results showed significantly shorter planning depths and higher response noise for older adults. Age differences in planning depth were only partially explained by well-known cognitive covariates such as working memory and processing speed. Consistent with previous findings, this indicates age-related shifts away from model-based behaviour in older adults. In addition to a shorter planning depth, our findings suggest that older adults also apply a variety of heuristical low-cost strategies.

Computational Mechanisms of Addiction and Anxiety: A Developmental Perspective

A central goal of computational psychiatry is to identify systematic relationships between transdiagnostic dimensions of psychiatric symptomatology and the latent learning and decision-making computations that inform individuals' thoughts, feelings, and choices. Most psychiatric disorders emerge prior to adulthood, yet little work has extended these computational approaches to study the development of psychopathology. Here, we lay out a roadmap for future studies implementing this approach by developing empirically and theoretically informed hypotheses about how developmental changes in model-based control of action and Pavlovian learning processes may modulate vulnerability to anxiety and addiction. We highlight how insights from studies leveraging computational approaches to characterize the normative developmental trajectories of clinically relevant learning and decision-making processes may suggest promising avenues for future developmental computational psychiatry research.

Negative symptoms, striatal dopamine and model-free reward decision-making in schizophrenia

Negative symptoms, such as lack of motivation or social withdrawal, are highly prevalent and debilitating in patients with schizophrenia. Underlying mechanisms of negative symptoms are incompletely understood, thereby preventing the development of targeted treatments. We hypothesized that in patients with schizophrenia during psychotic remission, impaired influences of both model-based and model-free reward predictions on decision-making ('reward prediction influence', RPI) underlie negative symptoms. We focused on psychotic remission, because psychotic symptoms might confound reward-based decision-making. Moreover, we hypothesized that impaired model-based/model-free RPIs depend on alterations of both associative striatum dopamine synthesis and storage (DSS) and executive functioning. Both factors influence RPI in healthy subjects and are typically impaired in schizophrenia. Twenty-five patients with schizophrenia with pronounced negative symptoms during psychotic remission and 24 healthy controls were included in the study. Negative symptom severity was measured by the Positive and Negative Syndrome Scale negative subscale, model-based/model-free RPI by the two-stage decision task, associative striatum DSS by 18F-DOPA positron emission tomography and executive functioning by the symbol coding task. Model-free RPI was selectively reduced in patients and associated with negative symptom severity as well as with reduced associative striatum DSS (in patients only) and executive functions (both in patients and controls). In contrast, model-based RPI was not altered in patients. Results provide evidence for impaired model-free reward prediction influence as a mechanism for negative symptoms in schizophrenia as well as for reduced associative striatum dopamine and executive dysfunction as relevant factors. Data suggest potential treatment targets for patients with schizophrenia and pronounced negative symptoms.

The Dopamine System in Mediating Alcohol Effects in Humans

Brain-imaging studies show that the development and maintenance of alcohol use disorder (AUD) is determined by a complex interaction of different neurotransmitter systems and multiple psychological factors. In this context, the dopaminergic reinforcement system appears to be of fundamental importance. We focus on the excitatory and depressant effects of acute versus chronic alcohol intake and its impact on dopaminergic neurotransmission. Furthermore, we describe alterations in dopaminergic neurotransmission as associated with symptoms of alcohol dependence. We specifically focus on neuroadaptations to chronic alcohol consumption and their effect on central processing of alcohol-associated and reward-related stimuli. Altered reward processing, complex conditioning processes, impaired reinforcement learning, and increased salience attribution to alcohol-associated stimuli enable alcohol cues to drive alcohol seeking and consumption. Finally, we will discuss how the neurobiological and neurochemical mechanisms of alcohol-associated alterations in reward processing and learning can interact with stress, cognition, and emotion processing.

Translational models of addiction phenotypes to advance addiction pharmacotherapy

Alcohol and substance use disorders are heterogeneous conditions with limited effective treatment options. While there have been prior attempts to classify addiction subtypes, they have not been translated into clinical practice. In an effort to better understand heterogeneity in psychiatric disorders, the National Institute for Mental Health Research Domain Criteria (RDoC) has challenged scientists to think beyond diagnostic symptoms and to consider the underlying features of psychopathology from a neuroscience-based framework. The field of addiction has grappled with this approach by considering several key constructs with the potential to capture RDoC domains. This critical review will focus on the efforts to apply translational models of addiction phenomenology in human clinical samples, including their relative strengths and weaknesses. Opportunities for forward and reverse translation are also discussed. Deep behavioral phenotyping using neuroscience-informed batteries shows promise for a better understanding of the clinical neuroscience of addiction and advancing precision medicine for alcohol and substance use disorders.

The modulation of acute stress on model-free and model-based reinforcement learning in gambling disorder

BACKGROUND AND AIMS

Experiencing acute stress is common in behavioral addictions such as gambling disorder. Additionally, like most substance-induced addictions, aberrant decision-making wherein a reactive habit-induced response (conceptualized as a Model-free [MF] in reinforcement learning) suppresses a flexible goal-directed response (conceptualized as a Model-based [MB]) is also common in gambling disorder. In the current study we investigated the influence of acute stress on the balance between habitual response and the goal-directed system.

METHODS

A sample of N = 116 problem gamblers (PG) and healthy controls (HC) performed an acute stress task - the Socially Evaluated Cold pressure task (SECPT) - or a control task. Self-reported stress and salivary cortisol were collected as measures of acute stress. Following the SECPT, participants performed the Two-Step Markov Task to account for the relative contribution of MB and MF strategies. Additionally, verbal working memory and IQ measures were collected to account for their mediating effects on the orchestration between MB/MF and the impact of stress.

RESULTS

Both groups had comparable baseline and stress-induced cortisol response to the SECPT. Non-stressed PG displayed lower MB learning than HC. MANOVA and regression analyses showed a deleterious effect of stress-induced cortisol response on the orchestration between MB and MF learning in HC but not in PG. These effects remained when controlling for working memory and IQ.

DISCUSSION AND CONCLUSIONS

We found an abnormal pattern of modulation of stress on the orchestration between MB and MF learning among PG. Several interpretations and future research directions are discussed.

Effects of chess-based cognitive remediation training as therapy add-on in alcohol and tobacco use disorders: protocol of a randomised, controlled clinical fMRI trial

BACKGROUND

Alcohol and tobacco use disorders (AUD, TUD) are frequent, both worldwide and in the German population, and cognitive impairments are known to facilitate instances of relapse. Cognitive training has been proposed for enhancing cognitive functioning and possibly improving treatment outcome in mental disorders. However, these effects and underlying neurobiological mechanisms are not yet fully understood regarding AUD and TUD. Examining the effect of chess-based cognitive remediation training (CB-CRT) on neurobiological, neuropsychological and psychosocial aspects as well as treatment outcomes will provide insights into mechanisms underlying relapse and abstinence and might help to improve health behaviour in affected individuals if used as therapy add-on.

METHODS AND ANALYSIS

N=96 individuals with either AUD (N=48) or TUD (N=48) between 18 and 65 years of age will participate in a randomised, controlled clinical functional MRI (fMRI) trial. Two control groups will receive treatment as usual, that is, AUD treatment in a clinic, TUD outpatient treatment. Two therapy add-on groups will receive a 6-week CB-CRT as a therapy add-on. FMRI tasks, neurocognitive tests will be administered before and afterwards. All individuals will be followed up on monthly for 3 months. Endpoints include alterations in neural activation and neuropsychological task performance, psychosocial functioning, and relapse or substance intake. Regarding fMRI analyses, a general linear model will be applied, and t-tests, full factorial models and regression analyses will be conducted on the second level. Behavioural and psychometric data will be analysed using t-tests, regression analyses, repeated measures and one-way analyses of variance.

ETHICS AND DISSEMINATION

This study has been approved by the ethics committee of the medical faculty Mannheim of the University of Heidelberg (2017-647N-MA). The findings of this study will be presented at conferences and published in peer-reviewed journals.

TRIAL REGISTRATION

The study was registered in the Clinical Trials Register (trial identifier: NCT04057534 at clinicaltrials.gov).

Gambling Environment Exposure Increases Temporal Discounting but Improves Model-Based Control in Regular Slot-Machine Gamblers

Gambling disorder is a behavioral addiction that negatively impacts personal finances, work, relationships and mental health. In this pre-registered study (https://osf.io/5ptz9/) we investigated the impact of real-life gambling environments on two computational markers of addiction, temporal discounting and model-based reinforcement learning. Gambling disorder is associated with increased temporal discounting and reduced model-based learning. Regular gamblers (n = 30, DSM-5 score range 3-9) performed both tasks in a neutral (café) and a gambling-related environment (slot-machine venue) in counterbalanced order. Data were modeled using drift diffusion models for temporal discounting and reinforcement learning via hierarchical Bayesian estimation. Replicating previous findings, gamblers discounted rewards more steeply in the gambling-related context. This effect was positively correlated with gambling related cognitive distortions (pre-registered analysis). In contrast to our pre-registered hypothesis, model-based reinforcement learning was improved in the gambling context. Here we show that temporal discounting and model-based reinforcement learning are modulated in opposite ways by real-life gambling cue exposure. Results challenge aspects of habit theories of addiction, and reveal that laboratory-based computational markers of psychopathology are under substantial contextual control.

Slower Learning Rates from Negative Outcomes in Substance Use Disorder over a 1-Year Period and Their Potential Predictive Utility

Computational modelling is a promising approach to parse dysfunctional cognitive processes in substance use disorders (SUDs), but it is unclear how much these processes change during the recovery period. We assessed 1-year follow-up data on a sample of treatment-seeking individuals with one or more SUDs (alcohol, cannabis, sedatives, stimulants, hallucinogens, and/or opioids; N = 83) that were previously assessed at baseline within a prior computational modelling study. Relative to healthy controls (HCs; N = 48), these participants were found at baseline to show altered learning rates and less precise action selection while completing an explore-exploit decision-making task. Here we replicated these analyses when these individuals returned and re-performed the task 1 year later to assess the stability of baseline differences. We also examined whether baseline modelling measures could predict symptoms at follow-up. Bayesian and frequentist analyses indicated that: (a) group differences in learning rates were stable over time (posterior probability = 1); and (b) intra-class correlations (ICCs) between model parameters at baseline and follow-up were significant and ranged from small to moderate (.25 ≤ ICCs ≤ .54). Exploratory analyses also suggested that learning rates and/or information-seeking values at baseline were associated with substance use severity at 1-year follow-up in stimulant and opioid users (.36 ≤ rs ≤ .43). These findings suggest that learning dysfunctions are moderately stable during recovery and could correspond to trait-like vulnerability factors. In addition, computational measures at baseline had some predictive value for changes in substance use severity over time and could be clinically informative.

Explicit knowledge of task structure is a primary determinant of human model-based action

Explicit information obtained through instruction profoundly shapes human choice behaviour. However, this has been studied in computationally simple tasks, and it is unknown how model-based and model-free systems, respectively generating goal-directed and habitual actions, are affected by the absence or presence of instructions. We assessed behaviour in a variant of a computationally more complex decision-making task, before and after providing information about task structure, both in healthy volunteers and in individuals suffering from obsessive-compulsive or other disorders. Initial behaviour was model-free, with rewards directly reinforcing preceding actions. Model-based control, employing predictions of states resulting from each action, emerged with experience in a minority of participants, and less in those with obsessive-compulsive disorder. Providing task structure information strongly increased model-based control, similarly across all groups. Thus, in humans, explicit task structural knowledge is a primary determinant of model-based reinforcement learning and is most readily acquired from instruction rather than experience.

Alcohol dependence decreases functional activation of the caudate nucleus during model-based decision processes

BACKGROUND

Impaired decision making, a key characteristic of alcohol dependence (AD), manifests in continuous alcohol consumption despite severe negative consequences. The neural basis of this impairment in individuals with AD and differences with known neural decision mechanisms among healthy subjects are not fully understood. In particular, it is unclear whether the choice behavior among individuals with AD is based on a general impairment of decision mechanisms or is mainly explained by altered value attribution, with an overly high subjective value attributed to alcohol-related stimuli.

METHODS

Here, we use a functional magnetic resonance imaging (fMRI) monetary reward task to compare the neural processes of model-based decision making and value computation between AD individuals (n = 32) and healthy controls (n = 32). During fMRI, participants evaluated monetary offers with respect to dynamically changing constraints and different levels of uncertainty.

RESULTS

Individuals with AD showed lower activation associated with model-based decision processes in the caudate nucleus than controls, but there were no group differences in value-related neural activity or task performance.

CONCLUSIONS

Our findings highlight the role of the caudate nucleus in impaired model-based decisions of alcohol-dependent individuals.

Psychoanalytical Considerations of Emotion Regulation Disorders in Multiple Complex-Traumatized Children—A Study Protocol of the Prospective Study MuKi

Studies in adults with mental disorders suggest that the experience of early and chronic trauma is associated with changes in reward expectancy and processing. In addition, severe childhood trauma has been shown to contribute to the development of mental disorders in general. Data on effects of early childhood trauma on reward expectancy and processing in middle childhood currently appear insufficient. The present study aims to fill this research gap by examining the effects of developmental trauma disorder (DTD) on reward expectancy and processing in children aged 8–12 years, testing the hypothesis that children with multiple complex traumas exhibit altered reward processing as a result of prior disappointing reward experiences. One main feature of developmental trauma disorder is early experiences of multiple separation from important and close relationships alongside other experiences of emotional or physical harm. In the sequel children often show affect regulation disorders. To investigate this, we have developed an adapted version of the Monetary Incentive Delay (MID) Task, which examines children’s expectation of reward or frustration. In this first study, behavioral data will be collected from N = 40 children (n = 20 experimental group and n = 20 healthy controls) using this adapted version of the MID Task. Children in the experimental group will be recruited from youth welfare centers in Frankfurt a.M., Germany. Healthy control subjects will be recruited from after-school-care facilities. A brief trauma screening will be conducted for both groups, experimental and control. If children show signs of trauma, the presence of a developmental trauma disorder will be further delineated by a German translation of the Developmental Trauma Disorder Structured Interview for Children (DTDSI-C) which was translated the first time in German by our research group. We hypothesize that children in the experimental group will be less accurate in performing the Monetary Incentive Delay Task because of their impaired emotion regulation skills due to emotional avoidance following developmental trauma. If the results of our initial behavioral study are promising, the MID task will be used in a future study to elucidate the relationship between trauma developmental disorder, reward expectancy and processing, and neurobiological processes in middle childhood.

Cortical control of striatal fast-spiking interneuron synchrony

KEY POINTS

Electrical synapses between striatal fast-spiking interneurons in adult mice occur in ∼8% of assayed pairs. Coincident, convergent cortical input onto fast-spiking interneurons significantly contributes to fast-spiking interneuron synchrony Electrical synapses between fast-spiking interneurons provide only minor enhancement of fast-spiking interneuron synchrony. These results suggest a mechanism by which adult mouse fast-spiking interneurons of the striatum synchronize in the face of declining expression of the electrical synapse-forming connexin-36 protein.

ABSTRACT

Inhibitory fast-spiking interneurons in the dorsal striatum regulate actions and action strategies, including habits. Fast-spiking interneurons are widely believed to synchronize their firing due to the electrical synapses formed between these neurons. However, neuronal modeling data suggest convergent cortical input may also drive synchrony in fast-spiking interneuron networks. To better understand how fast-spiking interneuron synchrony arises, we performed dual whole-cell patch clamp electrophysiology experiments to inform a simple Bayesian network modeling cortico-fast-spiking interneuron circuitry. Dual whole-cell patch clamp electrophysiology revealed that while responsivity to corticostriatal input activation was high in fast-spiking interneurons, few of these neurons exhibited electrical coupling in adult mice. In simulations of a cortico-fast-spiking interneuron network informed by these data, the degree of glutamatergic cortical convergence onto fast-spiking interneurons significantly increased fast-spiking interneuron synchronization while manipulations of electrical coupling between these neurons exerted relatively little impact. These results suggest that the primary source of functional coordination of fast-spiking interneuron activity in adulthood arises from convergent corticostriatal input activation. Abstract figure legend Dual whole-cell patch clamp recordings of dorsal striatal fast-spiking interneurons (FSIs; red circles) rarely (8 percentage) form electrical synapses with other FSIs in adult mouse. In a two-layer in silico model of cortical pyramidal neuron (gray triangles) input to FSIs using empirically defined cortico-FSI synaptic weights, synchronous FSI-FSI activity (in the absence of abundant electrical synapses) is achievable by convergent cortical pyramidal excitation of FSIs. This article is protected by copyright. All rights reserved.

Reinforcement learning detuned in addiction: integrative and translational approaches

Suboptimal decision-making strategies have been proposed to contribute to the pathophysiology of addiction. Decision-making, however, arises from a collection of computational components that can independently influence behavior. Disruptions in these different components can lead to decision-making deficits that appear similar behaviorally, but differ at the computational, and likely the neurobiological, level. Here, we discuss recent studies that have used computational approaches to investigate the decision-making processes underlying addiction. Studies in animal models have found that value updating following positive, but not negative, outcomes is predictive of drug use, whereas value updating following negative, but not positive, outcomes is disrupted following drug self-administration. We contextualize these findings with studies on the circuit and biological mechanisms of decision-making to develop a framework for revealing the biobehavioral mechanisms of addiction.

Dopamine enhances model-free credit assignment through boosting of retrospective model-based inference

Dopamine is implicated in representing model-free (MF) reward prediction errors a as well as influencing model-based (MB) credit assignment and choice. Putative cooperative interactions between MB and MF systems include a guidance of MF credit assignment by MB inference. Here, we used a double-blind, placebo-controlled, within-subjects design to test an hypothesis that enhancing dopamine levels boosts the guidance of MF credit assignment by MB inference. In line with this, we found that levodopa enhanced guidance of MF credit assignment by MB inference, without impacting MF and MB influences directly. This drug effect correlated negatively with a dopamine-dependent change in purely MB credit assignment, possibly reflecting a trade-off between these two MB components of behavioural control. Our findings of a dopamine boost in MB inference guidance of MF learning highlight a novel DA influence on MB-MF cooperative interactions.

Examining the effect of depressive symptoms on habit formation and habit-breaking

BACKGROUND AND OBJECTIVES

Dysfunction in reward processing is a hallmark feature of depression. In the context of reinforcement learning, previous research has linked depression with reliance on simple habit-driven ('model-free') learning strategies over more complex, goal-directed ('model-based') strategies. However, the relationship between depression and habit-breaking remains an under-explored research area. The current study sought to bridge this gap by investigating the effect of depressive symptoms on habit formation and habit-breaking under monetary and social feedback conditions. Additionally, we examined whether spontaneous eyeblink rate (EBR), an indirect marker for striatal dopamine levels, would modulate such effects.

METHODS

Depressive symptoms were operationalized using self-report measures. To examine differences in habit formation and habit breaking, undergraduate participants (N = 156) completed a two-stage reinforcement learning task with a devaluation procedure using either monetary or social feedback.

RESULTS

Regression results showed that in the monetary feedback condition, spontaneous EBR moderated the relationship between depressive symptoms and model-free strategies; individuals with more depressive symptomatology and high EBR (higher dopamine levels) exhibited increased reliance on model-free strategies. Depressive symptoms negatively predicted devaluation sensitivity, indicative of difficulty in habit-breaking, in both monetary and social feedback contexts.

LIMITATIONS

Social feedback relied on fixed feedback rather than real-time peer evaluations; depressive symptoms were measured using self-report rather than diagnostic criteria for Major Depressive Disorder; dopaminergic functioning was measured using EBR rather than PET imaging; potential confounds were not controlled for.

CONCLUSIONS

These findings have implications for identifying altered patterns of habit formation and deficits in habit-breaking among those experiencing depressive symptoms.

The Straw That Broke the Camel's Back: Natural Variations in 17β-Estradiol and COMT-Val158Met Genotype Interact in the Modulation of Model-Free and Model-Based Control

The sex hormone estradiol has recently gained attention in human decision-making research. Animal studies have already shown that estradiol promotes dopaminergic transmission and thus supports reward-seeking behavior and aspects of addiction. In humans, natural variations of estradiol across the menstrual cycle modulate the ability to learn from direct performance feedback ("model-free" learning). However, it remains unclear whether estradiol also influences more complex "model-based" contributions to reinforcement learning. Here, 41 women were tested twice - in the low and high estradiol state of the follicular phase of their menstrual cycle - with a Two-Step decision task designed to separate model-free from model-based learning. The results showed that in the high estradiol state women relied more heavily on model-free learning, and accomplished reduced performance gains, particularly during the more volatile periods of the task that demanded increased learning effort. In contrast, model-based control remained unaltered by the influence of hormonal state across the group. Yet, when accounting for individual differences in the genetic proxy of the COMT-Val158Met polymorphism (rs4680), we observed that only the participants homozygote for the methionine allele (n = 12; with putatively higher prefrontal dopamine) experienced a decline in model-based control when facing volatile reward probabilities. This group also showed the increase in suboptimal model-free control, while the carriers of the valine allele remained unaffected by the rise in endogenous estradiol. Taken together, these preliminary findings suggest that endogenous estradiol may affect the balance between model-based and model-free control, and particularly so in women with a high prefrontal baseline dopamine capacity and in situations of increased environmental volatility.

Fronto-striatal structures related with model-based control as an endophenotype for obsessive-compulsive disorder

Recent theories suggest a shift from model-based goal-directed to model-free habitual decision-making in obsessive–compulsive disorder (OCD). However, it is yet unclear, whether this shift in the decision process is heritable. We investigated 32 patients with OCD, 27 unaffected siblings (SIBs) and 31 healthy controls (HCs) using the two-step task. We computed behavioral and reaction time analyses and fitted a computational model to assess the balance between model-based and model-free control. 80 subjects also underwent structural imaging. We observed a significant ordered effect for the shift towards model-free control in the direction OCD > SIB > HC in our computational parameter of interest. However less directed analyses revealed no shift towards model-free control in OCDs. Nonetheless, we found evidence for reduced model-based control in OCDs compared to HCs and SIBs via 2nd stage reaction time analyses. In this measure SIBs also showed higher levels of model-based control than HCs. Across all subjects these effects were associated with the surface area of the left medial/right dorsolateral prefrontal cortex. Moreover, correlations between bilateral putamen/right caudate volumes and these effects varied as a function of group: they were negative in SIBs and OCDs, but positive in HCs. Associations between fronto-striatal regions and model-based reaction time effects point to a potential endophenotype for OCD.

Reduced Glutamate Turnover in the Putamen Is Linked With Automatic Habits in Human Cocaine Addiction

BACKGROUND

The balance between goal-directed behavior and habits has been hypothesized to be biased toward the latter in individuals with cocaine use disorder (CUD), suggesting possible neurochemical changes in the putamen, which may contribute to their compulsive behavior.

METHODS

We assessed habitual behavior in 48 patients with CUD and 42 healthy control participants using a contingency degradation paradigm and the Creature of Habit Scale. In a subgroup of this sample (CUD: n = 21; control participants: n = 22), we also measured glutamate and glutamine concentrations in the left putamen using ultra-high-field (7T) magnetic resonance spectroscopy. We hypothesized that increased habitual tendencies in patients with CUD would be associated with abnormal glutamatergic metabolites in the putamen.

RESULTS

Compared with their non-drug-using peers, patients with CUD exhibited greater habitual tendencies during contingency degradation, which correlated with increased levels of self-reported daily habits. We further identified a significant reduction in glutamate concentration and glutamate turnover (glutamate-to-glutamine ratio) in the putamen in patients with CUD, which was significantly related to the level of self-reported daily habits.

CONCLUSIONS

Patients with CUD exhibit enhanced habitual behavior, as assessed both by questionnaire and by a laboratory paradigm of contingency degradation. This automatic habitual tendency is related to a reduced glutamate turnover in the putamen, suggesting a dysregulation of habits caused by chronic cocaine use.

Model-Based and Model-Free Control Predicts Alcohol Consumption Developmental Trajectory in Young Adults: A 3-Year Prospective Study

BACKGROUND

A shift from goal-directed toward habitual control has been associated with alcohol dependence. Whether such a shift predisposes to risky drinking is not yet clear. We investigated how goal-directed and habitual control at age 18 predict alcohol use trajectories over the course of 3 years.

METHODS

Goal-directed and habitual control, as informed by model-based (MB) and model-free (MF) learning, were assessed with a two-step sequential decision-making task during functional magnetic resonance imaging in 146 healthy 18-year-old men. Three-year alcohol use developmental trajectories were based on either a consumption score from the self-reported Alcohol Use Disorders Identification Test (assessed every 6 months) or an interview-based binge drinking score (grams of alcohol/occasion; assessed every year). We applied a latent growth curve model to examine how MB and MF control predicted the drinking trajectory.

RESULTS

Drinking behavior was best characterized by a linear trajectory. MB behavioral control was negatively associated with the development of the binge drinking score; MF reward prediction error blood oxygen level-dependent signals in the ventromedial prefrontal cortex and the ventral striatum predicted a higher starting point and steeper increase of the Alcohol Use Disorders Identification Test consumption score over time, respectively.

CONCLUSIONS

We found that MB behavioral control was associated with the binge drinking trajectory, while the MF reward prediction error signal was closely linked to the consumption score development. These findings support the idea that unbalanced MB and MF control might be an important individual vulnerability in predisposing to risky drinking behavior.

Addiction history moderates the effect of prefrontal 10-Hz transcranial alternating current stimulation on habitual action selection

Individuals with substance use disorders (SUDs) transition more quickly from goal-directed to habitual action-selection, but the neural mechanisms underlying this phenomenon remain unclear. Data from animal models suggest that drugs of abuse can modify the neurocircuits that regulate action-selection, enhancing circuits that drive inflexible, habit-based stimulus-response (S-R) action-selection and weakening circuits that drive flexible, goal-directed actions. Here, we tested the effect of bilateral 10-Hz transcranial alternating current stimulation (10Ηz-tACs) of the dorsolateral prefrontal cortex on action-selection in men and women with a SUD history and an age- and sex-matched control group. We tested the hypothesis that true 10Ηz-tACS versus active sham stimulation would reduce perseverative errors after changed response contingencies for well-learned S-R associations, reflecting reduced habit-based action-selection, specifically in the SUD group. We found that 10 Hz-tACS increased perseverative errors in the control group, but in the SUD group, 10 Hz-tACS effects on perseverative errors depended on substance abuse duration: a longer addiction history was associated with a greater reduction of perseverative errors. These results suggest that 10Ηz-tACs altered circuit level dynamics regulating behavioral flexibility, and provide a foundation for future studies to test stimulation site, frequency, and timing specificity. Moreover, these data suggest that chronic substance abuse is associated with altered circuit dynamics that are ameliorated by 10Ηz-tACs. Determining the generalizability of these effects and their duration merits investigation as a direction for novel therapeutic interventions. These findings are timely based on growing interest in transcranial stimulation methods for treating SUDs.NEW & NOTEWORTHY Treating the executive dysfunction associated with addiction is hampered by redundancies in pharmacological regulation of different behavioral control circuits. Thus, nonpharmacological interventions hold promise for addiction treatment. Here, we show that, among people with an addiction history, 10-Hz transcranial alternating current stimulation (10Hz-tACS) of the dorsolateral prefrontal cortex can reduce habitual actions. The fact that 10Hz-tACS can regulate behavioral flexibility suggests its possible utility in reducing harmful habitual actions.

Instrumental and Pavlovian Mechanisms in Alcohol Use Disorder

Purpose of Review

Current theories of alcohol use disorders (AUD) highlight the importance of Pavlovian and instrumental learning processes mainly based on preclinical animal studies. Here, we summarize available evidence for alterations of those processes in human participants with AUD with a focus on habitual versus goal-directed instrumental learning, Pavlovian conditioning, and Pavlovian-to-instrumental transfer (PIT) paradigms.

Recent Findings

The balance between habitual and goal-directed control in AUD participants has been studied using outcome devaluation or sequential decision-making procedures, which have found some evidence of reduced goal-directed/model-based control, but little evidence for stronger habitual responding. The employed Pavlovian learning and PIT paradigms have shown considerable differences regarding experimental procedures, e.g., alcohol-related or conventional reinforcers or stimuli.

Summary

While studies of basic learning processes in human participants with AUD support a role of Pavlovian and instrumental learning mechanisms in the development and maintenance of drug addiction, current studies are characterized by large variability regarding methodology, sample characteristics, and results, and translation from animal paradigms to human research remains challenging. Longitudinal approaches with reliable and ecologically valid paradigms of Pavlovian and instrumental processes, including alcohol-related cues and outcomes, are warranted and should be combined with state-of-the-art imaging techniques, computational approaches, and ecological momentary assessment methods.

Mapping data-driven individualized neurobehavioral phenotypes in heavy alcohol drinkers

OBJECTIVE

Recent studies have examined the factor structure and associated correlates of three neurofunctional domains, executive function, incentive salience, and negative emotionality in the development and maintenance of alcohol use disorders in clinical samples. The current study sought to replicate and extend prior work by testing this 3-factor model, utilizing both exact and similar phenotypic measures, as well as novel measures, in a non-treatment-seeking sample.

METHODS

Self-report measures of alcohol addiction, impulsivity, behavior, and exposure to early-life stress were collected as part of baseline assessments for alcohol imaging and pharmacotherapy studies in 335 individuals. Confirmatory factor analysis (CFA) was used to examine model structure and fit. A multiple indicators, multiple causes (MIMIC) model identified predictors of latent factors identified by CFA.

RESULTS

Results supported an intercorrelated model with three factors: executive function, incentive salience, and emotionality. All factors were associated with current AUD, and incentive salience was uniquely associated with past 30-day drinking frequency. MIMIC results identified multiple significant predictors of these latent factors, including history of alcohol use disorder, positive family history of alcohol dependence, earlier age of first drink, and a history of childhood emotional abuse and physical neglect.

CONCLUSIONS

Our results support an intercorrelated 3-factor model of neurofunctional domains in alcohol use models, consistent with published findings. Because childhood physical neglect was a significant predictor of all latent factors, these results also highlight the significant negative impact of childhood neglect on later addiction development.

The effects of fatty acid amide hydrolase inhibition and monoacylglycerol lipase inhibition on habit formation in mice

Emerging data indicate that endocannabinoid signaling is critical to the formation of habitual behavior. Previous work demonstrated that antagonism of cannabinoid receptor type 1 (CB1R) with AM251 during operant training impairs habit formation, but it is not known if this behavioral effect is specific to disrupted signaling of the endocannabinoid ligands anandamide or 2-arachidonoyl glycerol (2-AG). Here, we used selective pharmacological compounds during operant training to determine the impact of fatty acid amide hydrolase (FAAH) inhibition to increase anandamide (and other n-acylethanolamines) or monoacylglycerol lipase (MAGL) inhibition to increase 2-AG levels on the formation of habitual behaviors in mice using a food-reinforced contingency degradation procedure. We found, contrary to our hypothesis, that inhibition of FAAH and of MAGL disrupted the formation of habits. Next, AM251 was administered during training to verify that impaired habit formation could be assessed using contingency degradation. AM251-exposed mice responded at lower rates during training and at higher rates in the test. To understand the inconsistency with published data, we performed a proof-of-principle dose-response experiment to compare AM251 in our vehicle-solution to the published vehicle-suspension on response rates. We found consistent reductions in response rate with increasing doses of AM251 in solution and an inconsistent dose-response relationship with AM251 in suspension. Together, our data suggest that further characterization of the role of CB1R signaling in the formation of habitual responding is warranted and that augmenting endocannabinoids may have clinical utility for prophylactically preventing aberrant habit formation such as that hypothesized to occur in substance use disorders.

Habitual Routines and Automatic Tendencies Differential Roles in Alcohol Misuse Among Undergraduates

There is a debate over whether actions that resist devaluation (i.e., compulsive alcohol consumption) are primarily habit- or goal-directed. The incentive habit account of compulsive actions has received support from behavioral paradigms and brain imaging. In addition, the self-reported Creature of Habit Scale (COHS) has been proposed to capture inter-individual differences in habitual tendencies. It is subdivided into two dimensions: routine and automaticity. We first considered a French version of this questionnaire for validation, based on a sample of 386 undergraduates. The relationship between two dimensions of habit and the risk of substance use disorder and impulsive personality traits was also investigated. COHS has good psychometric properties with both features of habits positively associated with an Obsessive-Compulsive Inventory score. Besides, the propensity to rely more on routines was associated with lower levels of alcohol abuse and nicotine use, suggesting that some degree of routine might act as a protective factor against substance use. In contrast, a high automaticity score was associated with an increased risk of harmful alcohol use. These results demonstrate that the COHS is a valid measure of habitual tendencies and represents a useful tool for capturing inter-individual variations in drug use problems in undergraduates.

The Anterior Cingulate Cortex Predicts Future States to Mediate Model-Based Action Selection

Behavioral control is not unitary. It comprises parallel systems, model based and model free, that respectively generate flexible and habitual behaviors. Model-based decisions use predictions of the specific consequences of actions, but how these are implemented in the brain is poorly understood. We used calcium imaging and optogenetics in a sequential decision task for mice to show that the anterior cingulate cortex (ACC) predicts the state that actions will lead to, not simply whether they are good or bad, and monitors whether outcomes match these predictions. ACC represents the complete state space of the task, with reward signals that depend strongly on the state where reward is obtained but minimally on the preceding choice. Accordingly, ACC is necessary only for updating model-based strategies, not for basic reward-driven action reinforcement. These results reveal that ACC is a critical node in model-based control, with a specific role in predicting future states given chosen actions.

Recent Developments in the Habit Hypothesis of OCD and Compulsive Disorders

This chapter aims to familiarise the reader with a diverse and fast-growing literature concerning the role that habits play in obsessive-compulsive disorder (OCD). Core concepts will be introduced, including how the balance between habits and a more deliberate form of action selection (goal-directed control) has traditionally been measured and how cross-species translation, neuroscience tools, and computational modelling have been used to build on these basic principles and reveal core mechanisms under study today. Next, the application of these methods to the study of OCD and related disorders will be detailed, converging on a theory that enhanced habit expression, and indeed compulsions in OCD, might arise from deficits in goal-directed control systems. These basic findings will be contextualised in terms of major tide changes in the field, including the shift from categorical disease frameworks to dimensional ones. Mechanistically, recent research concerning how goal-directed deficits arise, perhaps through failures in the construction of a mental model, are discussed along with studies critically evaluating our ability to measure habits in humans, in a laboratory setting. The chapter ends with a nod to the future, focusing on the need for clinically oriented, longitudinal, and intervention-based research that aim to translate what is now a wealth of cross-sectional mechanistic insights to the clinic.

Computational theory-driven studies of reinforcement learning and decision-making in addiction: What have we learned?

Computational psychiatry provides a powerful new approach for linking the behavioral manifestations of addiction to their precise cognitive and neurobiological substrates. However, this emerging area of research is still limited in important ways. While research has identified features of reinforcement learning and decision-making in substance users that differ from health, less emphasis has been placed on capturing addiction cycles/states dynamically, within-person. In addition, the focus on few behavioral variables at a time has precluded more detailed consideration of related processes and heterogeneous clinical profiles. We propose that a longitudinal and multidimensional examination of value-based processes, a type of dynamic "computational fingerprint", will provide a more complete understanding of addiction as well as aid in developing better tailored and timed interventions.

Neurostimulation Reveals Context-Dependent Arbitration Between Model-Based and Model-Free Reinforcement Learning

While it is established that humans use model-based (MB) and model-free (MF) reinforcement learning in a complementary fashion, much less is known about how the brain determines which of these systems should control behavior at any given moment. Here we provide causal evidence for a neural mechanism that acts as a context-dependent arbitrator between both systems. We applied excitatory and inhibitory transcranial direct current stimulation over a region of the left ventrolateral prefrontal cortex previously found to encode the reliability of both learning systems. The opposing neural interventions resulted in a bidirectional shift of control between MB and MF learning. Stimulation also affected the sensitivity of the arbitration mechanism itself, as it changed how often subjects switched between the dominant system over time. Both of these effects depended on varying task contexts that either favored MB or MF control, indicating that this arbitration mechanism is not context-invariant but flexibly incorporates information about current environmental demands.

Risk and Protective Factors for Alcohol Use Disorders Across the Lifespan

Purpose of Review

Losing and regaining control over alcohol intake varies as a function of individual-level predictors across the lifespan. Specifically, the interplay of protective and risk factors for losing and regaining control, particularly in real-life settings, is thus far poorly understood. Individual differences in cognition, affect, emotion regulation, social factors, and personality traits, together with individual differences in brain structure and function, and biological markers of stress exposure may have different effects on alcohol consumption in different age groups. We will review current evidence for age-specific effects for losing and regaining control over alcohol intake and propose a framework for investigation across age groups.

Recent Findings

We find evidence for differences in relative impact of psychosocial predictors of alcohol consumption as a function of age that varies by gender. There is theoretical reason to assume that predictors vary in the time course of their taking effect: While e.g., early trauma and personality traits may be conceptualized as more distant antecedents of alcohol consumption, cognition, affect and emotion regulation can be conceptualized as co-correlates, where variation over periods of months may go along with changes in alcohol consumption. At the same time, craving, current stressors, and priming events may serve as short-term or immediate causes of alcohol consumption.

Summary

We propose a combination of longitudinal age cohorts to (i) identify individual-level differences (using latent growth curve models) and profiles (using latent growth mixture models) of the psychosocial and biological variables of interest that predict regaining or losing control, and ambulatory assessments every 2 days, in order to (ii) investigate effects of triggers and risk factors on current alcohol consumption. This approach will allow us to characterize age-related differences in the interplay between these factors in real-life settings.

Goal-Directed and Habitual Control in Smokers

Introduction

Harmful behavior such as smoking may reflect a disturbance in the balance of goal-directed and habitual control. Animal models suggest that habitual control develops after prolonged substance use. In this study, we investigated whether smokers (N = 49) differ from controls (N = 46) in the regulation of goal-directed and habitual behavior. It was also investigated whether individual differences in nicotine dependence levels were associated with habitual responding.

Methods

We used two different multistage instrumental learning tasks that consist of an instrumental learning phase, subsequent outcome devaluation, and a testing phase to measure the balance between goal-directed and habitual responding. The testing phases of these tasks occurred after either appetitive versus avoidance instrumental learning. The appetitive versus aversive instrumental learning stages in the two different tasks modeled positive versus negative reinforcement, respectively.

Results

Smokers and nonsmoking controls did not differ on habitual versus goal-directed control in either task. Individual differences in nicotine dependence within the group of smokers, however, were positively associated with habitual responding after appetitive instrumental learning. This effect seems to be due to impaired stimulus-outcome learning, thereby hampering goal-directed task performance and tipping the balance to habitual responding.

Conclusions

The current finding highlights the importance of individual differences within smokers. For future research, neuroimaging studies are suggested to further unravel the nature of the imbalance between goal-directed versus habitual control in severely dependent smokers by directly measuring activity in the corresponding brain systems.

Implications

Goal-directed versus habitual behavior in substance use and addiction is highly debated. This study investigated goal-directed versus habitual control in smokers. The findings suggest that smokers do not differ from controls in goal-directed versus habitual control. Individual differences in nicotine dependence within smokers, however, were positively associated with habitual responding after appetitive instrumental learning. This effect seems to be due to impaired stimulus-outcome learning, thereby hampering goal-directed task performance and tipping the balance to habitual responding. These findings add to the ongoing debate on habitual versus goal-directed control in addiction and emphasize the importance of individual differences within smokers.

Behavioral, neurobiological, and neurochemical mechanisms of ethanol self-administration: A translational review

Alcohol use disorder has multiple characteristics including excessive ethanol consumption, impaired control over drinking behaviors, craving and withdrawal symptoms, compulsive seeking behaviors, and is considered a chronic condition. Relapse is common. Determining the neurobiological targets of ethanol and the adaptations induced by chronic ethanol exposure is critical to understanding the clinical manifestation of alcohol use disorders, the mechanisms underlying the various features of the disorder, and for informing medication development. In the present review, we discuss ethanol's interactions with a variety of neurotransmitter systems, summarizing findings from preclinical and translational studies to highlight recent progress in the field. We then describe animal models of ethanol self-administration, emphasizing the value, limitations, and validity of commonly used models. Lastly, we summarize the behavioral changes induced by chronic ethanol self-administration, with an emphasis on cue-elicited behavior, the role of ethanol-related memories, and the emergence of habitual ethanol seeking behavior.

Computational models of drug use and addiction: A review

In this brief review, we describe current computational models of drug-use and addiction that fall into 2 broad categories: mathematically based models that rely on computational theories, and brain-based models that link computations to brain areas or circuits. Across categories, many are models of learning and decision-making, which may be compromised in addiction. Several mathematical models take predictive coding approaches, focusing on Bayesian prediction error. Other models focus on learning processes and (traditional) prediction error. Brain-based models have incorporated prefrontal cortex, basal ganglia, and the dopamine system, based on the effects of drugs on dopamine, motivation, and executive control circuits. Several models specifically describe how behavioral control may transition from habitual to goal-directed systems, consistent with computational accounts of compromised "model-based" control. Some brain-based models have linked this to the transition of behavioral control from ventral to dorsal striatum. Overall, we propose that while computational models capture some aspects of addiction and have advanced our thinking, most have focused on the effects of drug use rather than addiction per se, most have not been tested on and/or supported by human data, and few capture multiple stages and symptoms of addiction. We conclude by suggesting a path forward for computational models of addiction. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Combined model-free and model-sensitive reinforcement learning in non-human primates

Contemporary reinforcement learning (RL) theory suggests that potential choices can be evaluated by strategies that may or may not be sensitive to the computational structure of tasks. A paradigmatic model-free (MF) strategy simply repeats actions that have been rewarded in the past; by contrast, model-sensitive (MS) strategies exploit richer information associated with knowledge of task dynamics. MF and MS strategies should typically be combined, because they have complementary statistical and computational strengths; however, this tradeoff between MF/MS RL has mostly only been demonstrated in humans, often with only modest numbers of trials. We trained rhesus monkeys to perform a two-stage decision task designed to elicit and discriminate the use of MF and MS methods. A descriptive analysis of choice behaviour revealed directly that the structure of the task (of MS importance) and the reward history (of MF and MS importance) significantly influenced both choice and response vigour. A detailed, trial-by-trial computational analysis confirmed that choices were made according to a combination of strategies, with a dominant influence of a particular form of model sensitivity that persisted over weeks of testing. The residuals from this model necessitated development of a new combined RL model which incorporates a particular credit assignment weighting procedure. Finally, response vigor exhibited a subtly different collection of MF and MS influences. These results provide new illumination onto RL behavioural processes in non-human primates.

We routinely solve planning problems in which present decisions have consequences in the future. These pose complex computational and statistical problems and are addressed by multiple systems in the brain which use different solutions to these problems, and which may compete and cooperate. We trained two rhesus monkeys on a paradigmatic planning task that transparently reveals canonical aspects of different strategies. We performed a detailed behavioral analysis using methods of reinforcement learning on choice and reaction time to reveal conjoint influences and structural interactions of different sources of information. We show the strengths and limitations of these analyses, at the same time as we provide a novel perspective on how different learning systems interact for choice in non-human primates.

Alcohol Hangover Does Not Alter the Application of Model-Based and Model-Free Learning Strategies

Frequent alcohol binges shift behavior from goal-directed to habitual processing modes. This shift in reward-associated learning strategies plays a key role in the development and maintenance of alcohol use disorders and seems to persist during (early stages of) sobriety in at-risk drinkers. Yet still, it has remained unclear whether this phenomenon might be associated with alcohol hangover and thus also be found in social drinkers. In an experimental crossover design, n = 25 healthy young male participants performed a two-step decision-making task once sober and once hungover (i.e., when reaching sobriety after consuming 2.6 g of alcohol per estimated liter of total body water). This task allows the separation of effortful model-based and computationally less demanding model-free learning strategies. The experimental induction of alcohol hangover was successful, but we found no significant hangover effects on model-based and model-free learning scores, the balance between model-free and model-based valuation (ω), or perseveration tendencies (π). Bayesian analyses provided positive evidence for the null hypothesis for all measures except π (anecdotal evidence for the null hypothesis). Taken together, alcohol hangover, which results from a single binge drinking episode, does not impair the application of effortful and computationally costly model-based learning strategies and/or increase model-free learning strategies. This supports the notion that the behavioral deficits observed in at-risk drinkers are most likely not caused by the immediate aftereffects of individual binge drinking events.

Endocannabinoid contributions to alcohol habits and motivation: Relevance to treatment

Individuals with alcohol use disorder exhibit compulsive habitual behaviors that are thought to be, in part, a consequence of chronic and persistent use of alcohol. The endocannabinoid system plays a critical role in habit learning and in ethanol self-administration, but the role of this neuromodulatory system in the expression of habitual alcohol seeking is unknown. Here, we investigated the role of the endocannabinoid system in established alcohol habits using contingency degradation in male C57BL/6 mice. We found that administration of the novel diacyl glycerol lipase inhibitor DO34, which decreases the biosynthesis of the endocannabinoid 2-arachidonoyl glycerol (2-AG), reduced habitual responding for ethanol and ethanol approach behaviors. Moreover, administration of the endocannabinoid transport inhibitor AM404 or the cannabinoid receptor type 1 antagonist AM251 produced similar reductions in habitual responding for ethanol and ethanol approach behaviors. Notably, AM404 was also able to reduce ethanol seeking and consumption in mice that were insensitive to lithium chloride-induced devaluation of ethanol. Conversely, administration of JZL184, a monoacyl glycerol lipase inhibitor that increases levels of 2-AG, increased motivation to respond for ethanol on a progressive ratio schedule of reinforcement. These results demonstrate an important role for endocannabinoid signaling in the motivation to seek ethanol, in ethanol-motivated habits, and suggest that pharmacological manipulations of endocannabinoid signaling could be effective therapeutics for treating alcohol use disorder.