Computational modelling reveals contrasting effects on reinforcement learning and cognitive flexibility in stimulant use disorder and obsessive-compulsive disorder: remediating effects of dopaminergic D2/3 receptor agents

Atomoxetine Reduces Decisional Impulsivity in Human Cocaine Addiction

BACKGROUND

Impulsivity is a well-known determinant of maladaptive behavior in cocaine use disorder (CUD), but there are currently no effective strategies for managing excessive impulsivity. Growing evidence from preclinical and clinical studies suggests that atomoxetine, a selective noradrenaline reuptake inhibitor, is effective in improving impulse control in both healthy individuals and individuals with neuropsychiatric conditions.

METHODS

We investigated the effects of atomoxetine on decisional impulsivity in patients with CUD. In a randomized, double-blind, placebo-controlled, crossover study, 28 patients diagnosed with moderate-to-severe CUD and 28 matched healthy control participants completed the Cambridge Gambling Task in 2 separate sessions, where they received either placebo or a single dose of 40 mg atomoxetine at each session. Computational modeling was used to break down decision making into 3 separable components: value, probability, and decisional impulsivity.

RESULTS

Our analyses revealed that patients with CUD were impaired in all components of decision making. Atomoxetine selectively reduced decisional impulsivity in patients with CUD by reducing their risk-seeking tendencies while enhancing their ability to tolerate delays. By contrast, atomoxetine did not affect impulsivity in control participants, but increased their sensitivity to prospective losses.

CONCLUSIONS

Taken together, our findings support the hypothesis of noradrenergic dysfunction in patients with CUD and provide novel translational evidence for the efficacy of atomoxetine in remediating decisional impulsivity in CUD.

A generalized reward processing deficit pathway to negative symptoms across diagnostic boundaries

BACKGROUND

Negative symptoms are a key feature of several psychiatric disorders. Difficulty identifying common neurobiological mechanisms that cut across diagnostic boundaries might result from equifinality (i.e., multiple mechanistic pathways to the same clinical profile), both within and across disorders. This study used a data-driven approach to identify unique subgroups of participants with distinct reward processing profiles to determine which profiles predicted negative symptoms.

METHODS

Participants were a transdiagnostic sample of youth from a multisite study of psychosis risk, including 110 individuals at clinical high-risk for psychosis (CHR; meeting psychosis-risk syndrome criteria), 88 help-seeking participants who failed to meet CHR criteria and/or who presented with other psychiatric diagnoses, and a reference group of 66 healthy controls. Participants completed clinical interviews and behavioral tasks assessing four reward processing constructs indexed by the RDoC Positive Valence Systems: hedonic reactivity, reinforcement learning, value representation, and effort-cost computation.

RESULTS

k-means cluster analysis of clinical participants identified three subgroups with distinct reward processing profiles, primarily characterized by: a value representation deficit (54%), a generalized reward processing deficit (17%), and a hedonic reactivity deficit (29%). Clusters did not differ in rates of clinical group membership or psychiatric diagnoses. Elevated negative symptoms were only present in the generalized deficit cluster, which also displayed greater functional impairment and higher psychosis conversion probability scores.

CONCLUSIONS

Contrary to the equifinality hypothesis, results suggested one global reward processing deficit pathway to negative symptoms independent of diagnostic classification. Assessment of reward processing profiles may have utility for individualized clinical prediction and treatment.

Dissociating social reward learning and behavior in alcohol use disorder

BACKGROUND

Alcohol use disorder (AUD) is associated with deficits in social cognition and behavior, but why these deficits are acquired is unknown. We hypothesized that a reduced association between actions and outcomes for others, i.e., social reward learning, would explain prevalent social deficiencies in AUD.

METHODS

We conducted one laboratory study (n = 234) and one confirmatory online study (n = 258), comparing young adults with AUD to age-, gender-, and education-matched healthy controls on a standardized reward learning task. In the task, participants learned to maximize reward for another person and for oneself. To elucidate the potential relation between reward learning and social behavior in AUD, we administered two measures: a dictator game task and a self-report measure. Finally, we applied reinforcement learning models to examine the computational properties of learning.

RESULTS

Social and individual learning, expressed in choice behavior, were comparable in individuals with AUD and healthy controls. Individual differences in learning were not associated with reduced social behavior in AUD. Computational modeling suggested that the learning mechanisms are comparable in AUD and healthy controls and indifferent to whether learning maximizes reward for another person or oneself.

CONCLUSIONS

Individuals with AUD demonstrated preserved reward learning abilities that do not vary with social behavior. Together, these results indicate that reward processes may not be relevant for understanding compromised social behavior in AUD.

Impaired arbitration between reward-related decision-making strategies in Alcohol Users compared to Alcohol Non-Users: a computational modeling study

Reinforcement learning studies propose that decision-making is guided by a tradeoff between computationally cheaper model-free (habitual) control and costly model-based (goal-directed) control. Greater model-based control is typically used under highly rewarding conditions to minimize risk and maximize gain. Although prior studies have shown impairments in sensitivity to reward value in individuals with frequent alcohol use, it is unclear how these individuals arbitrate between model-free and model-based control based on the magnitude of reward incentives. In this study, 81 individuals (47 frequent Alcohol Users and 34 Alcohol Non-Users) performed a modified 2-step learning task where stakes were sometimes high, and other times they were low. Maximum a posteriori fitting of a dual-system reinforcement-learning model was used to assess the degree of model-based control, and a utility model was used to assess risk sensitivity for the low- and high-stakes trials separately. As expected, Alcohol Non-Users showed significantly higher model-based control in higher compared to lower reward conditions, whereas no such difference between the two conditions was observed for the Alcohol Users. Additionally, both groups were significantly less risk-averse in higher compared to lower reward conditions. However, Alcohol Users were significantly less risk-averse compared to Alcohol Non-Users in the higher reward condition. Lastly, greater model-based control was associated with a less risk-sensitive approach in Alcohol Users. Taken together, these results suggest that frequent Alcohol Users may have impaired metacontrol, making them less flexible to varying monetary rewards and more prone to risky decision-making, especially when the stakes are high.

Mapping computational cognitive profiles of aging to dissociable brain and sociodemographic factors

Aging is associated with declines in cognition and brain structural integrity. However, there is equivocality over (1) the specificity of affected domains in different people, (2) the location of associated patterns of brain structural deterioration, and (3) the sociodemographic factors contributing to 'unhealthy' cognition. We aimed to identify cognitive profiles displayed by older adults and determine brain and sociodemographic features potentially shaping these profiles. A sample of Southeast-Asian older adults (N = 386) participated in a multi-session study comprising cognitive testing, neuroimaging, and a structured interview. We used computational models to extract latent mechanisms underlying cognitive flexibility and response inhibition. Data-driven methods were used to construct cognitive profiles based on standard performance measures and model parameters. We also investigated grey matter volume and machine-learning derived 'brain-ages'. A profile associated with poor set-shifting and rigid focusing was associated with widespread grey matter reduction in cognitive control regions. A slow responding profile was associated with advanced brain-age. Both profiles were correlated with poor socioeconomic standing and cognitive reserve. We found that the impact of sociodemographic factors on cognitive profiles was partially mediated by total grey and white matter, and dorsolateral prefrontal and cerebellar volumes. This study furthers understanding of how distinct aging profiles of cognitive impairment uniquely correspond to specific vs. global brain deterioration and the significance of socioeconomic factors in informing cognitive performance in older age.

Compulsion is associated with impaired goal-directed and habitual learning and responding in obsessive-compulsive disorder

Background

Previous research has found that compulsions in obsessive-compulsive disorder (OCD) are associated with an imbalance between goal-directed and habitual responses. However, the cognitive mechanisms underlying how goal-directed and habitual behaviors are learned, and how these learning deficits affect the response process, remain unclear. The present study aimed to investigate these cognitive mechanisms and examine how they were involved in the mechanism of compulsions.

Methods

A total of 49 patients with OCD and 38 healthy controls (HCs) were recruited to perform the revised "slip of action test". A reinforcement learning model was constructed, and model parameters including learning rates, reinforcement sensitivity, and perseveration were estimated using a hierarchical Bayesian approach. Comparisons of these parameters were made between the OCD group and HCs, and the associations with performance during the outcome devalued stage and clinical presentations were assessed.

Results

In the outcome devalued stage, patients with OCD exhibited greatet responsiveness to the devalued outcome, indicating their impairment in flexible and goal-directed behavioral control. Computational modeling further revealed that, during the instrumental learning stage, patients with OCD showed reduced learning rates, decreased perseveration, and heightened reinforcement sensitivity as compared with HCs. The learning rate and perseveration during instrumental learning were significantly correlated with the performance in the outcome devalued stage and compulsive scores in OCD.

Conclusions

The results indicate that patients with OCD exhibit deficits in updating the associative strength based on prediction errors and are more likely to doubt established correct associations during goal-directed and habitual learning. These deficits may contribute to the inflexible goal-directed behavioral control and are involved in the mechanism of compulsion in OCD.

Reduced brain connectivity underlying value-based choices and outcomes in stimulant use disorder

BACKGROUND

Patients with stimulant use disorder (SUD) show impairments when making value-based choices that are associated with disruptions in neural processing across brain networks. Making optimal choices requires learning from outcomes to update knowledge and further optimise ongoing behaviour. The optimal functioning of neural systems that underpin the ability to make favourable choices is an essential component for life functioning, and successful recovery in patients with SUD. Therefore, we sought to investigate the neural processes that underpin value-based choices in SUD patients. We hypothesise that patients with SUD have reduced functional connectivity while making financial choices during a probabilistic reinforcement learning task.

METHODS

We investigated connectivity associated with loss and reward value-based choices and their outcomes in patients with SUD and healthy control participants during a pharmacological magnetic resonance imaging study. Participants received a single dose of a dopamine receptor agonist, pramipexole, and a dopamine receptor antagonist, amisulpride, in a randomised, double-blind, placebo-controlled, balanced, crossover design. Functional task-related connectivity was analysed taking a whole brain connectomics approach to identify networks that are differentially modulated by dopaminergic receptor functioning.

RESULTS

SUD patients showed widespread reductions in connectivity during both reward and loss value-based choices and outcomes, which were negatively correlated with the duration of stimulant drug use. Disturbances to functional brain connectivity in SUD patients during task performance were not modulated acutely by either amisulpride or pramipexole.

CONCLUSIONS

Reductions in brain connectivity, particularly when making value-based choices and processing outcomes, may underlie learning impairments in SUD patients. Given that acute dopaminergic modulation did not improve brain connectivity impairments in SUD patients, it is likely that alternative treatments are needed.

Surprising sounds influence risky decision making

Adaptive behavior depends on appropriate responses to environmental uncertainty. Incidental sensory events might simply be distracting and increase errors, but alternatively can lead to stereotyped responses despite their irrelevance. To evaluate these possibilities, we test whether task-irrelevant sensory prediction errors influence risky decision making in humans across seven experiments (total n = 1600). Rare auditory sequences preceding option presentation systematically increase risk taking and decrease choice perseveration (i.e., increased tendency to switch away from previously chosen options). The risk-taking and perseveration effects are dissociable by manipulating auditory statistics: when rare sequences end on standard tones, including when rare sequences consist only of standard tones, participants are less likely to perseverate after rare sequences but not more likely to take risks. Computational modeling reveals that these effects cannot be explained by increased decision noise but can be explained by value-independent risky bias and perseveration parameters, decision biases previously linked to dopamine. Control experiments demonstrate that both surprise effects can be eliminated when tone sequences are presented in a balanced or fully predictable manner, and that surprise effects cannot be explained by erroneous beliefs. These findings suggest that incidental sounds may influence many of the decisions we make in daily life.

5-HT 2A and 5-HT 2C receptor antagonism differentially modulate reinforcement learning and cognitive flexibility: behavioural and computational evidence

RATIONALE

Cognitive flexibility, the ability to adapt behaviour in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive-compulsive disorder and major depressive disorder. Evidence suggests that flexibility, which can be operationalised using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behaviour and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown.

OBJECTIVES

We investigated the effects of 5-HT2A receptor (5-HT2AR) and 5-HT2C receptor (5-HT2CR) antagonism on flexibility and underlying RL mechanisms.

METHODS

Thirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT2AR and 5-HT2CR antagonists M100907 and SB-242084, respectively, on reversal learning and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task.

RESULTS

5-HT2AR antagonism impaired reversal learning only after an initial perseverative phase, during a period of random choice and then new learning. 5-HT2CR antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT2AR antagonism decreased punishment learning rate (i.e. negative feedback) at high and low doses. The low dose also decreased reinforcement sensitivity (beta) and increased stimulus and side stickiness (i.e., the tendency to repeat a choice regardless of outcome). 5-HT2CR antagonism also decreased beta, but reduced side stickiness.

CONCLUSIONS

These data indicate that 5-HT2A and 5-HT2CRs both modulate different aspects of flexibility, with 5-HT2ARs modulating learning from negative feedback as measured using RL parameters and 5-HT2CRs for learning from positive feedback assessed through conventional measures.

Compulsive avoidance in youths and adults with OCD: an aversive pavlovian-to-instrumental transfer study

Compulsive behaviour may often be triggered by Pavlovian cues. Assessing how Pavlovian cues drive instrumental behaviour in obsessive-compulsive disorder (OCD) is therefore crucial to understand how compulsions develop and are maintained. An aversive Pavlovian-to-Instrumental transfer (PIT) paradigm, particularly one involving avoidance/cancellation of negative outcomes, can enable such investigation and has not previously been studied in clinical-OCD. Forty-one participants diagnosed with OCD (21 adults; 20 youths) and 44 controls (21 adults; 23 youths) completed an aversive PIT task. Participants had to prevent the delivery of unpleasant noises by moving a joystick in the correct direction. They could infer these correct responses by learning appropriate response-outcome (instrumental) and stimulus-outcome (Pavlovian) associations. We then assessed whether Pavlovian cues elicited specific instrumental avoidance responses (specific PIT) and induced general instrumental avoidance (general PIT). We investigated whether task learning and confidence indices influenced PIT strength differentially between groups. There was no overall group difference in PIT performance, although youths with OCD showed weaker specific PIT than youth controls. However, urge to avoid unpleasant noises and preference for safe over unsafe stimuli influenced specific and general PIT respectively in OCD, while PIT in controls was more influenced by confidence in instrumental and Pavlovian learning. Thus, in OCD, implicit motivational factors, but not learnt knowledge, may contribute to the successful integration of aversive Pavlovian and instrumental cues. This implies that compulsive avoidance may be driven by these automatic processes. Youths with OCD show deficits in specific PIT, suggesting cue integration impairments are only apparent in adolescence. These findings may be clinically relevant as they emphasise the importance of targeting such implicit motivational processes when treating OCD.

Action sequence learning, habits, and automaticity in obsessive-compulsive disorder

This study investigates the goal/habit imbalance theory of compulsion in obsessive-compulsive disorder (OCD), which postulates enhanced habit formation, increased automaticity, and impaired goal/habit arbitration. It directly tests these hypotheses using newly developed behavioral tasks. First, OCD patients and healthy participants were trained daily for a month using a smartphone app to perform chunked action sequences. Despite similar procedural learning and attainment of habitual performance (measured by an objective automaticity criterion) by both groups, OCD patients self-reported higher subjective habitual tendencies via a recently developed questionnaire. Subsequently, in a re-evaluation task assessing choices between established automatic and novel goal-directed actions, both groups were sensitive to re-evaluation based on monetary feedback. However, OCD patients, especially those with higher compulsive symptoms and habitual tendencies, showed a clear preference for trained/habitual sequences when choices were based on physical effort, possibly due to their higher attributed intrinsic value. These patients also used the habit-training app more extensively and reported symptom relief post-study. The tendency to attribute higher intrinsic value to familiar actions may be a potential mechanism leading to compulsions and an important addition to the goal/habit imbalance hypothesis in OCD. We also highlight the potential of smartphone app training as a habit reversal therapeutic tool.

Ketamine produces no detectable long-term positive or negative effects on cognitive flexibility or reinforcement learning of male rats

RATIONALE

Patients with major depressive disorder (MDD) often experience abnormalities in behavioral adaptation following environmental changes (i.e., cognitive flexibility) and tend to undervalue positive outcomes but overvalue negative outcomes. The probabilistic reversal learning task (PRL) is used to study these deficits across species and to explore drugs that may have therapeutic value. Selective serotonin-reuptake inhibitors (SSRIs) have limited effectiveness in treating MDD and produce inconsistent effects in non-human versions of the PRL. As such, ketamine, a novel and potentially rapid-acting therapeutic, has begun to be examined using the PRL. Two previous studies examining the effects of ketamine in the PRL have shown conflicting results and only examined short-term effects of ketamine.

OBJECTIVE

This experiment examined PRL performance across a 2-week period following a single exposure to a ketamine dose that varied across groups.

METHODS

After five sessions of PRL training, groups of rats received an injection of either 0, 10, 20 or 30 mg/kg ketamine. One-hour post-injection, rats engaged in the PRL, and subsequently sessions continued daily for 2 weeks. Traditional behavioral and computational reinforcement learning-derived measures were examined.

RESULTS

Results showed that ketamine had acute effects 1-h post-injection, including a significant decrease in the value of the punishment learning rate. Beyond 1 h, ketamine produced no detectable improvements nor decrements in performance across 2 weeks.

CONCLUSION

Overall, the present results suggest that the range of ketamine doses examined do not have long-term positive or negative effects on cognitive flexibility or reward processing in healthy rats as measured by the PRL.

The empirical status of predictive coding and active inference

Research on predictive processing models has focused largely on two specific algorithmic theories: Predictive Coding for perception and Active Inference for decision-making. While these interconnected theories possess broad explanatory potential, they have only recently begun to receive direct empirical evaluation. Here, we review recent studies of Predictive Coding and Active Inference with a focus on evaluating the degree to which they are empirically supported. For Predictive Coding, we find that existing empirical evidence offers modest support. However, some positive results can also be explained by alternative feedforward (e.g., feature detection-based) models. For Active Inference, most empirical studies have focused on fitting these models to behavior as a means of identifying and explaining individual or group differences. While Active Inference models tend to explain behavioral data reasonably well, there has not been a focus on testing empirical validity of active inference theory per se, which would require formal comparison to other models (e.g., non-Bayesian or model-free reinforcement learning models). This review suggests that, while promising, a number of specific research directions are still necessary to evaluate the empirical adequacy and explanatory power of these algorithms.

Pramipexole Enhances Reward Learning by Preserving Value Estimates

BACKGROUND

Dopamine D2-like agonists show promise as treatments for depression. They are thought to act by enhancing reward learning; however, the mechanisms by which they achieve this are not clear. Reinforcement learning accounts describe 3 distinct candidate mechanisms: increased reward sensitivity, increased inverse decision-temperature, and decreased value decay. As these mechanisms produce equivalent effects on behavior, arbitrating between them requires measurement of how expectations and prediction errors are altered. We characterized the effects of 2 weeks of the D2-like agonist pramipexole on reward learning and used functional magnetic resonance imaging measures of expectation and prediction error to assess which of these 3 mechanistic processes were responsible for the behavioral effects.

METHODS

Forty healthy volunteers (50% female) were randomized to 2 weeks of pramipexole (titrated to 1 mg/day) or placebo in a double-blind, between-subject design. Participants completed a probabilistic instrumental learning task before and after the pharmacological intervention, with functional magnetic resonance imaging data collected at the second visit. Asymptotic choice accuracy and a reinforcement learning model were used to assess reward learning.

RESULTS

Pramipexole increased choice accuracy in the reward condition with no effect on losses. Participants who received pramipexole had increased blood oxygen level-dependent response in the orbital frontal cortex during the expectation of win trials but decreased blood oxygen level-dependent response to reward prediction errors in the ventromedial prefrontal cortex. This pattern of results indicates that pramipexole enhances choice accuracy by reducing the decay of estimated values during reward learning.

CONCLUSIONS

The D2-like receptor agonist pramipexole enhances reward learning by preserving learned values. This is a plausible mechanism for pramipexole's antidepressant effect.

Compulsivity and Inhibitory Control Deficits in Abstinent Individuals With Heroin Addiction and Their Biological Siblings Compared With Unrelated Healthy Control Participants

BACKGROUND

Compulsivity represents the performance of persistent and repetitive acts despite negative consequences and is considered one of the critical mechanisms for drug addiction. Although compulsivity-related neurocognitive impairments have been linked to addiction, it remains unclear whether these deficits might have predated drug abuse as potential familial susceptibilities.

METHODS

A large sample of 213 adult participants were recruited, including 70 abstinent individuals addicted to heroin (HAs), 69 unaffected biological siblings of the HAs (siblings), and 74 unrelated healthy control participants. Compulsivity-related neurocognitive functions were evaluated using the intradimensional/extradimensional set-shift task and a probabilistic reversal learning task. Compulsive traits were measured by the Obsessive-Compulsive Inventory-Revised. Inhibitory control was assessed using the stop signal task and Stroop Color and Word Test. Network models for group recognition were conducted using multilayer perceptron neural networks.

RESULTS

Data indicated that both HAs and siblings performed worse than healthy control participants on compulsivity-related aspects (i.e., shifting and reversal learning functions) and inhibitory control and had higher levels of self-reported compulsive traits. Furthermore, neural models revealed that a possible 3-facet clustering of neurocognitive deficits was linked to both HAs and siblings.

CONCLUSIONS

Our findings suggest that deficits in shift reversal and inhibitory control aspects and elevated compulsive traits, shared by HAs and their unaffected siblings, may putatively represent conceivable markers associated with familial vulnerabilities implicated in the development of heroin dependence.

Comparable roles for serotonin in rats and humans for computations underlying flexible decision-making

Serotonin is critical for adapting behavior flexibly to meet changing environmental demands. Cognitive flexibility is important for successful attainment of goals, as well as for social interactions, and is frequently impaired in neuropsychiatric disorders, including obsessive-compulsive disorder. However, a unifying mechanistic framework accounting for the role of serotonin in behavioral flexibility has remained elusive. Here, we demonstrate common effects of manipulating serotonin function across two species (rats and humans) on latent processes supporting choice behavior during probabilistic reversal learning, using computational modelling. The findings support a role of serotonin in behavioral flexibility and plasticity, indicated, respectively, by increases or decreases in choice repetition ('stickiness') or reinforcement learning rates following manipulations intended to increase or decrease serotonin function. More specifically, the rate at which expected value increased following reward and decreased following punishment (reward and punishment 'learning rates') was greatest after sub-chronic administration of the selective serotonin reuptake inhibitor (SSRI) citalopram (5 mg/kg for 7 days followed by 10 mg/kg twice a day for 5 days) in rats. Conversely, humans given a single dose of an SSRI (20 mg escitalopram), which can decrease post-synaptic serotonin signalling, and rats that received the neurotoxin 5,7-dihydroxytryptamine (5,7-DHT), which destroys forebrain serotonergic neurons, exhibited decreased reward learning rates. A basic perseverative tendency ('stickiness'), or choice repetition irrespective of the outcome produced, was likewise increased in rats after the 12-day SSRI regimen and decreased after single dose SSRI in humans and 5,7-DHT in rats. These common effects of serotonergic manipulations on rats and humans-identified via computational modelling-suggest an evolutionarily conserved role for serotonin in plasticity and behavioral flexibility and have clinical relevance transdiagnostically for neuropsychiatric disorders.

Perseveration and Shifting in Obsessive-Compulsive Disorder as a Function of Uncertainty, Punishment, and Serotonergic Medication

Background

The nature of cognitive flexibility deficits in obsessive-compulsive disorder (OCD), which historically have been tested with probabilistic reversal learning tasks, remains elusive. Here, a novel deterministic reversal task and inclusion of unmedicated patients in the study sample illuminated the role of fixed versus uncertain rules/contingencies and of serotonergic medication. Additionally, our understanding of probabilistic reversal was enhanced through theoretical computational modeling of cognitive flexibility in OCD.

Methods

We recruited 49 patients with OCD, 21 of whom were unmedicated, and 43 healthy control participants matched for age, IQ, and gender. Participants were tested on 2 tasks: a novel visuomotor deterministic reversal learning task with 3 reversals (feedback rewarding/punishing/neutral) measuring accuracy/perseveration and a 2-choice visual probabilistic reversal learning task with uncertain feedback and a single reversal measuring win-stay and lose-shift. Bayesian computational modeling provided measures of learning rate, reinforcement sensitivity, and stimulus stickiness.

Results

Unmedicated patients with OCD were impaired on the deterministic reversal task under punishment only at the first and third reversals compared with both control participants and medicated patients with OCD, who had no deficit. Perseverative errors were correlated with OCD severity. On the probabilistic reversal task, unmedicated patients were only impaired at reversal, whereas medicated patients were impaired at both the learning and reversal stages. Computational modeling showed that the overall change was reduced feedback sensitivity in both OCD groups.

Conclusions

Both perseveration and increased shifting can be observed in OCD, depending on test conditions including the predictability of reinforcement. Perseveration was related to clinical severity and remediated by serotonergic medication.

Neurobehavioral Precursors of Compulsive Cocaine Seeking in Dual Frontostriatal Circuits

Background

Only some individuals who use drugs recreationally eventually develop a substance use disorder, characterized in part by the rigid engagement in drug foraging behavior (drug seeking), which is often maintained in the face of adverse consequences (i.e., is compulsive). The neurobehavioral determinants of this individual vulnerability have not been fully elucidated.

Methods

Using a prospective longitudinal study involving 39 male rats, we combined multidimensional characterization of behavioral traits of vulnerability to stimulant use disorder (impulsivity and stickiness) and resilience (sign tracking and sensation seeking/locomotor reactivity to novelty) with magnetic resonance imaging to identify the structural and functional brain correlates of the later emergence of compulsive drug seeking in drug-naïve subjects. We developed a novel behavioral procedure to investigate the individual tendency to persist in drug-seeking behavior in the face of punishment in a drug-free state in subjects with a prolonged history of cocaine seeking under the control of the conditioned reinforcing properties of a drug-paired Pavlovian conditioned stimulus.

Results

In drug-naïve rats, the tendency to develop compulsive cocaine seeking was characterized by behavioral stickiness-related functional hypoconnectivity between the prefrontal cortex and posterior dorsomedial striatum in combination with impulsivity-related structural alterations in the infralimbic cortex, anterior insula, and nucleus accumbens.

Conclusions

These findings show that the vulnerability to developing compulsive cocaine-seeking behavior stems from preexisting structural or functional changes in two distinct corticostriatal systems that underlie deficits in impulse control and goal-directed behavior.

Computational modelling of reinforcement learning and functional neuroimaging of probabilistic reversal for dissociating compulsive behaviours in gambling and cocaine use disorders

BACKGROUND

Individuals with cocaine use disorder or gambling disorder demonstrate impairments in cognitive flexibility: the ability to adapt to changes in the environment. Flexibility is commonly assessed in a laboratory setting using probabilistic reversal learning, which involves reinforcement learning, the process by which feedback from the environment is used to adjust behavior.

AIMS

It is poorly understood whether impairments in flexibility differ between individuals with cocaine use and gambling disorders, and how this is instantiated by the brain. We applied computational modelling methods to gain a deeper mechanistic explanation of the latent processes underlying cognitive flexibility across two disorders of compulsivity.

METHOD

We present a re-analysis of probabilistic reversal data from individuals with either gambling disorder (n = 18) or cocaine use disorder (n = 20) and control participants (n = 18), using a hierarchical Bayesian approach. Furthermore, we relate behavioural findings to their underlying neural substrates through an analysis of task-based functional magnetic resonanceimaging (fMRI) data.

RESULTS

We observed lower 'stimulus stickiness' in gambling disorder, and report differences in tracking expected values in individuals with gambling disorder compared to controls, with greater activity during reward expected value tracking in the cingulate gyrus and amygdala. In cocaine use disorder, we observed lower responses to positive punishment prediction errors and greater activity following negative punishment prediction errors in the superior frontal gyrus compared to controls.

CONCLUSIONS

Using a computational approach, we show that individuals with gambling disorder and cocaine use disorder differed in their perseverative tendencies and in how they tracked value neurally, which has implications for psychiatric classification.

Genetic variation in the dopamine system is associated with mixed-strategy decision-making in patients with Parkinson's disease

Decision-making during mixed-strategy games requires flexibly adapting choice strategies in response to others' actions and dynamically tracking outcomes. Such decisions involve diverse cognitive processes, including reinforcement learning, which are affected by disruptions to the striatal dopamine system. We therefore investigated how genetic variation in dopamine function affected mixed-strategy decision-making in Parkinson's disease (PD), which involves striatal dopamine pathology. Sixty-six PD patients (ages 49-85, Hoehn and Yahr Stages 1-3) and 22 healthy controls (ages 54-75) competed in a mixed-strategy game where successful performance depended on minimizing choice biases (i.e., flexibly adapting choices trial by trial). Participants also completed a fixed-strategy task that was matched for sensory input, motor outputs and overall reward rate. Factor analyses were used to disentangle cognitive from motor aspects within both tasks. Using a within-subject, multi-centre design, patients were examined on and off dopaminergic therapy, and genetic variation was examined via a multilocus genetic profile score representing the additive effects of three single nucleotide polymorphisms (SNPs) that influence dopamine transmission: rs4680 (COMT Val158 Met), rs6277 (C957T) and rs907094 (encoding DARPP-32). PD and control participants displayed comparable mixed-strategy choice behaviour (overall); however, PD patients with genetic profile scores indicating higher dopamine transmission showed improved performance relative to those with low scores. Exploratory follow-up tests across individual SNPs revealed better performance in individuals with the C957T polymorphism, reflecting higher striatal D2/D3 receptor density. Importantly, genetic variation modulated cognitive aspects of performance, above and beyond motor function, suggesting that genetic variation in dopamine signalling may underlie individual differences in cognitive function in PD.

Active learning impairments in substance use disorders when resolving the explore-exploit dilemma: A replication and extension of previous computational modeling results

BACKGROUND

Substance use disorders (SUDs) represent a major public health risk. Yet, our understanding of the mechanisms that maintain these disorders remains incomplete. In a recent computational modeling study, we found initial evidence that SUDs are associated with slower learning rates from negative outcomes and less value-sensitive choice (low "action precision"), which could help explain continued substance use despite harmful consequences.

METHODS

Here we aimed to replicate and extend these results in a pre-registered study with a new sample of 168 individuals with SUDs and 99 healthy comparisons (HCs). We performed the same computational modeling and group comparisons as in our prior report (doi: 10.1016/j.drugalcdep.2020.108208) to confirm previously observed effects. After completing all pre-registered replication analyses, we then combined the previous and current datasets (N = 468) to assess whether differences were transdiagnostic or driven by specific disorders.

RESULTS

Replicating prior results, SUDs showed slower learning rates for negative outcomes in both Bayesian and frequentist analyses (partial η2=.02). Previously observed differences in action precision were not confirmed. Learning rates for positive outcomes were also similar between groups. Logistic regressions including all computational parameters as predictors in the combined datasets could differentiate several specific disorders from HCs, but could not differentiate most disorders from each other.

CONCLUSIONS

These results provide robust evidence that individuals with SUDs adjust behavior more slowly in the face of negative outcomes than HCs. They also suggest this effect is common across several different SUDs. Future research should examine its neural basis and whether learning rates could represent a new treatment target or moderator of treatment outcome.

Effect of lysergic acid diethylamide (LSD) on reinforcement learning in humans

BACKGROUND

The non-selective serotonin 2A (5-HT2A) receptor agonist lysergic acid diethylamide (LSD) holds promise as a treatment for some psychiatric disorders. Psychedelic drugs such as LSD have been suggested to have therapeutic actions through their effects on learning. The behavioural effects of LSD in humans, however, remain incompletely understood. Here we examined how LSD affects probabilistic reversal learning (PRL) in healthy humans.

METHODS

Healthy volunteers received intravenous LSD (75 μg in 10 mL saline) or placebo (10 mL saline) in a within-subjects design and completed a PRL task. Participants had to learn through trial and error which of three stimuli was rewarded most of the time, and these contingencies switched in a reversal phase. Computational models of reinforcement learning (RL) were fitted to the behavioural data to assess how LSD affected the updating ('learning rates') and deployment of value representations ('reinforcement sensitivity') during choice, as well as 'stimulus stickiness' (choice repetition irrespective of reinforcement history).

RESULTS

Raw data measures assessing sensitivity to immediate feedback ('win-stay' and 'lose-shift' probabilities) were unaffected, whereas LSD increased the impact of the strength of initial learning on perseveration. Computational modelling revealed that the most pronounced effect of LSD was the enhancement of the reward learning rate. The punishment learning rate was also elevated. Stimulus stickiness was decreased by LSD, reflecting heightened exploration. Reinforcement sensitivity differed by phase.

CONCLUSIONS

Increased RL rates suggest LSD induced a state of heightened plasticity. These results indicate a potential mechanism through which revision of maladaptive associations could occur in the clinical application of LSD.

A computational network dynamical modeling for abnormal oscillation and deep brain stimulation control of obsessive-compulsive disorder

Obsessive-compulsive disorder (OCD) is associated with multi-nodal abnormalities in brain networks, characterized by recurrent intrusive thoughts (obsessions) and repetitive behaviours or mental acts (compulsions), which might manifest as pathological low-frequency oscillations in the frontal EEG and low-frequency bursting firing patterns in the subthalamus nucleus (STN). Abnormalities in the cortical-striatal-thalamic-cortical (CSTC) loop, including dysregulation of serotonin, dopamine, and glutamate systems, are considered to contribute to certain types of OCD. Here, we extend a biophysical computational model to investigate the effect of orbitofronto-subcortical loop abnormalities on network oscillations. Particularly, the OCD lesion process is simulated by the loss of connectivity from striatal parvalbumin interneurons (PV) to medium spiny neurons (MSNs), excessive activation to the hyperdirect pathway, and high dopamine concentrations. By calculating low-frequency oscillation power in the STN, STN burst index, and average firing rates levels of the cortex and thalamus, we demonstrate that the model can explain the pathology of glutamatergic and dopamine system dysregulation, the effects of pathway imbalance, and neuropsychiatric treatment in OCD. In addition, results indicate the abnormal brain rhythms caused by the dysregulation of orbitofronto-subcortical loop may serve as a biomarker of OCD. Our studies can help to understand the cause of OCD, thereby facilitating the diagnosis of OCD and the development of new therapeutics.

Methylphenidate as a treatment option for substance use disorder: a transdiagnostic perspective

A transition in viewing mental disorders from conditions defined as a set of unique characteristics to one of the quantitative variations on a collection of dimensions allows overlap between disorders. The overlap can be utilized to extend to treatment approaches. Here, we consider the overlap between attention-deficit/hyperactivity disorder and substance use disorder to probe the suitability to use methylphenidate as a treatment for substance use disorder. Both disorders are characterized by maladaptive goal-directed behavior, impaired cognitive control, hyperactive phasic dopaminergic neurotransmission in the striatum, prefrontal hypoactivation, and reduced frontal cortex gray matter volume/density. In addition, methylphenidate has been shown to improve cognitive control and normalize associated brain activation in substance use disorder patients and clinical trials have found methylphenidate to improve clinical outcomes. Despite the theoretical basis and promising, but preliminary, outcomes, many questions remain unanswered. Most prominent is whether all patients who are addicted to different substances may equally profit from methylphenidate treatment.

Study protocol: How does cognitive flexibility relate to other executive functions and learning in healthy young adults?

BACKGROUND

Cognitive flexibility (CF) enables individuals to readily shift from one concept or mode of practice/thoughts to another in response to changes in the environment and feedback, making CF vital to optimise success in obtaining goals. However, how CF relates to other executive functions (e.g., working memory, response inhibition), mental abilities (e.g., creativity, literacy, numeracy, intelligence, structure learning), and social factors (e.g., multilingualism, tolerance of uncertainty, perceived social support, social decision-making) is less well understood. The current study aims to (1) establish the construct validity of CF in relation to other executive function skills and intelligence, and (2) elucidate specific relationships between CF, structure learning, creativity, career decision making and planning, and other life skills.

METHODS

This study will recruit up to 400 healthy Singaporean young adults (age 18-30) to complete a wide range of cognitive tasks and social questionnaires/tasks. The richness of the task/questionnaire battery and within-participant administration enables us to use computational modelling and structural equation modelling to examine connections between the latent constructs of interest.

SIGNIFICANCE AND IMPACT

The current study is the first systematic investigation into the construct validity of CF and its interrelationship with other important cognitive skills such as learning and creativity, within an Asian context. The study will further explore the concept of CF as a non-unitary construct, a novel theoretical proposition in the field. The inclusion of a structure learning paradigm is intended to inform future development of a novel intervention paradigm to enhance CF. Finally, the results of the study will be useful for informing classroom pedagogy and the design of lifelong learning policies and curricula, as part of the wider remit of the Cambridge-NTU Centre for Lifelong Learning and Individualised Cognition (CLIC).

Opponent learning with different representations in the cortico-basal ganglia pathways can develop obsession-compulsion cycle

Obsessive-compulsive disorder (OCD) has been suggested to be associated with impairment of model-based behavioral control. Meanwhile, recent work suggested shorter memory trace for negative than positive prediction errors (PEs) in OCD. We explored relations between these two suggestions through computational modeling. Based on the properties of cortico-basal ganglia pathways, we modeled human as an agent having a combination of successor representation (SR)-based system that enables model-based-like control and individual representation (IR)-based system that only hosts model-free control, with the two systems potentially learning from positive and negative PEs in different rates. We simulated the agent's behavior in the environmental model used in the recent work that describes potential development of obsession-compulsion cycle. We found that the dual-system agent could develop enhanced obsession-compulsion cycle, similarly to the agent having memory trace imbalance in the recent work, if the SR- and IR-based systems learned mainly from positive and negative PEs, respectively. We then simulated the behavior of such an opponent SR+IR agent in the two-stage decision task, in comparison with the agent having only SR-based control. Fitting of the agents' behavior by the model weighing model-based and model-free control developed in the original two-stage task study resulted in smaller weights of model-based control for the opponent SR+IR agent than for the SR-only agent. These results reconcile the previous suggestions about OCD, i.e., impaired model-based control and memory trace imbalance, raising a novel possibility that opponent learning in model(SR)-based and model-free controllers underlies obsession-compulsion. Our model cannot explain the behavior of OCD patients in punishment, rather than reward, contexts, but it could be resolved if opponent SR+IR learning operates also in the recently revealed non-canonical cortico-basal ganglia-dopamine circuit for threat/aversiveness, rather than reward, reinforcement learning, and the aversive SR + appetitive IR agent could actually develop obsession-compulsion if the environment is modeled differently.

Impairment of arbitration between model-based and model-free reinforcement learning in obsessive-compulsive disorder

Introduction

Obsessive-compulsive disorder (OCD) is characterized by an imbalance between goal-directed and habitual learning systems in behavioral control, but it is unclear whether these impairments are due to a single system abnormality of the goal-directed system or due to an impairment in a separate arbitration mechanism that selects which system controls behavior at each point in time.

Methods

A total of 30 OCD patients and 120 healthy controls performed a 2-choice, 3-stage Markov decision-making paradigm. Reinforcement learning models were used to estimate goal-directed learning (as model-based reinforcement learning) and habitual learning (as model-free reinforcement learning). In general, 29 high Obsessive-Compulsive Inventory-Revised (OCI-R) score controls, 31 low OCI-R score controls, and all 30 OCD patients were selected for the analysis.

Results

Obsessive-compulsive disorder (OCD) patients showed less appropriate strategy choices than controls regardless of whether the OCI-R scores in the control subjects were high (p = 0.012) or low (p < 0.001), specifically showing a greater model-free strategy use in task conditions where the model-based strategy was optimal. Furthermore, OCD patients (p = 0.001) and control subjects with high OCI-R scores (H-OCI-R; p = 0.009) both showed greater system switching rather than consistent strategy use in task conditions where model-free use was optimal.

Conclusion

These findings indicated an impaired arbitration mechanism for flexible adaptation to environmental demands in both OCD patients and healthy individuals reporting high OCI-R scores.

Active learning impairments in substance use disorders when resolving the explore-exploit dilemma: A replication and extension of previous computational modeling results

Background

Substance use disorders (SUDs) represent a major public health risk. Yet, our understanding of the mechanisms that maintain these disorders remains incomplete. In a recent computational modeling study, we found initial evidence that SUDs are associated with slower learning rates from negative outcomes and less value-sensitive choice (low "action precision"), which could help explain continued substance use despite harmful consequences.

Methods

Here we aimed to replicate and extend these results in a pre-registered study with a new sample of 168 individuals with SUDs and 99 healthy comparisons (HCs). We performed the same computational modeling and group comparisons as in our prior report (doi: 10.1016/j.drugalcdep.2020.108208) to confirm previously observed effects. After completing all pre-registered replication analyses, we then combined the previous and current datasets (N = 468) to assess whether differences were transdiagnostic or driven by specific disorders.

Results

Replicating prior results, SUDs showed slower learning rates for negative outcomes in both Bayesian and frequentist analyses (η 2 =.02). Previously observed differences in action precision were not confirmed. Logistic regressions including all computational parameters as predictors in the combined datasets could differentiate several specific disorders from HCs, but could not differentiate most disorders from each other.

Conclusions

These results provide robust evidence that individuals with SUDs have more difficulty adjusting behavior in the face of negative outcomes than HCs. They also suggest this effect is common across several different SUDs. Future research should examine its neural basis and whether learning rates could represent a new treatment target or moderator of treatment outcome.

Individuals with problem gambling and obsessive-compulsive disorder learn through distinct reinforcement mechanisms

Obsessive-compulsive disorder (OCD) and pathological gambling (PG) are accompanied by deficits in behavioural flexibility. In reinforcement learning, this inflexibility can reflect asymmetric learning from outcomes above and below expectations. In alternative frameworks, it reflects perseveration independent of learning. Here, we examine evidence for asymmetric reward-learning in OCD and PG by leveraging model-based functional magnetic resonance imaging (fMRI). Compared with healthy controls (HC), OCD patients exhibited a lower learning rate for worse-than-expected outcomes, which was associated with the attenuated encoding of negative reward prediction errors in the dorsomedial prefrontal cortex and the dorsal striatum. PG patients showed higher and lower learning rates for better- and worse-than-expected outcomes, respectively, accompanied by higher encoding of positive reward prediction errors in the anterior insula than HC. Perseveration did not differ considerably between the patient groups and HC. These findings elucidate the neural computations of reward-learning that are altered in OCD and PG, providing a potential account of behavioural inflexibility in those mental disorders.

Chronic escitalopram in healthy volunteers has specific effects on reinforcement sensitivity: a double-blind, placebo-controlled semi-randomised study

Several studies of the effects on cognition of selective serotonin reuptake inhibitors (SSRI), administered either acutely or sub-chronically in healthy volunteers, have found changes in learning and reinforcement outcomes. In contrast, to our knowledge, there have been no studies of chronic effects of escitalopram on cognition in healthy volunteers. This is important in view of its clinical use in major depressive disorder (MDD) and obsessive-compulsive disorder (OCD). Consequently, we aimed to investigate the chronic effect of the SSRI, escitalopram, on measures of 'cold' cognition (including inhibition, cognitive flexibility, memory) and 'hot cognition' including decision-making and particularly reinforcement learning. The study, conducted at the University of Copenhagen between May 2020 and October 2021, used a double-blind placebo-controlled design with 66 healthy volunteers, semi-randomised to receive either 20 mg of escitalopram (n = 32) or placebo (n = 34), balanced for age, sex and intelligence quotient (IQ) for at least 21 days. Questionnaires, neuropsychological tests and serum escitalopram measures were taken. We analysed group differences on the cognitive measures using linear regression models as well as innovative hierarchical Bayesian modelling of the Probabilistic Reversal Learning (PRL) task. The novel and important finding was that escitalopram reduced reinforcement sensitivity compared to placebo on both the Sequential Model-Based/Model-Free task and the PRL task. We found no other significant group differences on 'cold' or 'hot' cognition. These findings demonstrate that serotonin reuptake inhibition is involved in reinforcement learning in healthy individuals. Lower reinforcement sensitivity in response to chronic SSRI administration may reflect the 'blunting' effect often reported by patients with MDD treated with SSRIs. Trial Registration: NCT04239339 .

Sex-dependent effects of early life stress on reinforcement learning and limbic cortico-striatal functional connectivity

Major depressive disorder (MDD) is a stress-related condition hypothesized to involve aberrant reinforcement learning (RL) with positive and negative stimuli. The present study investigated whether repeated early maternal separation (REMS) stress, a procedure widely recognized to cause depression-like behaviour, affects how subjects learn from positive and negative feedback. The REMS procedure was implemented by separating male and female rats from their dam for 6 h each day from post-natal day 5-19. Control rat offspring were left undisturbed during this period. Rats were tested as adults for behavioral flexibility and feedback sensitivity on a probabilistic reversal learning task. A computational approach based on RL theory was used to derive latent behavioral variables related to reward learning and flexibility. To assess underlying brain substrates, a seed-based functional MRI connectivity analysis was applied both before and after an additional adulthood stressor in control and REMS rats. Female but not male rats exposed to REMS stress showed increased response 'stickiness' (repeated responses regardless of reward outcome). Following repeated adulthood stress, reduced functional connectivity from the basolateral amygdala (BLA) to the dorsolateral striatum (DLS), cingulate cortex (Cg), and anterior insula (AI) cortex was observed in females. By contrast, control male rats exposed to the second stressor showed impaired learning from negative feedback (i.e., non-reward) and reduced functional connectivity from the BLA to the DLS and AI compared to maternally separated males. RL in male rats exposed to REMS was unaffected. The fMRI data further revealed that connectivity between the mOFC and other prefrontal cortical and subcortical structures was positively correlated with response 'stickiness'. These findings reveal differences in how females and males respond to early life adversity and subsequent stress. These effects may be mediated by functional divergence in resting-state connectivity between the basolateral amygdala and fronto-striatal brain regions.

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders

In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.

Repeated eticlopride administration increases dopamine D2 receptor expression and restores behavioral flexibility disrupted by methamphetamine exposure to male rats

Repeated methamphetamine exposure impairs reversal learning in laboratory animals and downregulates dopamine D₂ receptor expression. In the present study, we tested the possibility that repeated exposure to the dopamine D₂ antagonist, eticlopride, would increase D₂ receptor expression, improve behavioral flexibility and restore behavioral flexibility that was disrupted by exposure to methamphetamine in rats. Male Sprague-Dawley rats received repeated daily pretreatment with the dopamine D₂ antagonist, eticlopride (0.0 or 0.3 mg/kg/day, 14 days). Three days after the last treatment, whole brain (minus olfactory bulbs and cerebellum) dopamine D₂ receptor expression was measured using flow cytometry in one group and reversal learning performance was measured in another group. Reversal learning was also measured in other groups prior to and after methamphetamine exposure (0.0 or 2.0 mg/kg, 4 injections, 2 h apart, 1 day) followed by repeated eticlopride (0.0 or 0.3 mg/kg, 14 days) treatment. Eticlopride treatment increased D₂ receptor expression and improved reversal learning performance. Methamphetamine impaired reversal learning performance and eticlopride treatment reversed the deficit. These results suggest that repeated administration of eticlopride can restore behavioral flexibility and that upregulation of D₂ receptors might be an effective adjunct to treatment of methamphetamine misuse.

Value-free random exploration is linked to impulsivity

Deciding whether to forgo a good choice in favour of exploring a potentially more rewarding alternative is one of the most challenging arbitrations both in human reasoning and in artificial intelligence. Humans show substantial variability in their exploration, and theoretical (but only limited empirical) work has suggested that excessive exploration is a critical mechanism underlying the psychiatric dimension of impulsivity. In this registered report, we put these theories to test using large online samples, dimensional analyses, and computational modelling. Capitalising on recent advances in disentangling distinct human exploration strategies, we not only demonstrate that impulsivity is associated with a specific form of exploration—value-free random exploration—but also explore links between exploration and other psychiatric dimensions.

The Stage 1 protocol for this Registered Report was accepted in principle on 19/03/2021. The protocol, as accepted by the journal, can be found at 10.6084/m9.figshare.14346506.v1.

Deciding between known rewarding options and exploring novel avenues is central to decision making. Humans show variability in their exploration. Here, the authors show that impulsivity is associated to an increased usage of a cognitively cheap (and sometimes sub-optimal) exploration strategy.

Reinforcement learning modeling reveals a reward-history-dependent strategy underlying reversal learning in squirrel monkeys

Insight into psychiatric disease and development of therapeutics relies on behavioral tasks that study similar cognitive constructs in multiple species. The reversal learning task is one popular paradigm that probes flexible behavior, aberrations of which are thought to be important in a number of disease states. Despite widespread use, there is a need for a high-throughput primate model that can bridge the genetic, anatomic, and behavioral gap between rodents and humans. Here, we trained squirrel monkeys, a promising preclinical model, on an image-guided deterministic reversal learning task. We found that squirrel monkeys exhibited two key hallmarks of behavior found in other species: integration of reward history over many trials and a side-specific bias. We adapted a reinforcement learning model and demonstrated that it could simulate squirrel monkey-like behavior, capture training-related trajectories, and provide insight into the strategies animals employed. These results validate squirrel monkeys as a model in which to study behavioral flexibility. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Computational analysis of probabilistic reversal learning deficits in male subjects with alcohol use disorder

BACKGROUND

Alcohol use disorder is characterized by perseverative alcohol use despite negative consequences. This hallmark feature of addiction potentially relates to impairments in behavioral flexibility, which can be measured by probabilistic reversal learning (PRL) paradigms. We here aimed to examine the cognitive mechanisms underlying impaired PRL task performance in patients with alcohol use disorder (AUDP) using computational models of reinforcement learning.

METHODS

Twenty-eight early abstinent AUDP and 27 healthy controls (HC) performed an extensive PRL paradigm. We compared conventional behavioral variables of choices (perseveration; correct responses) between groups. Moreover, we fitted Bayesian computational models to the task data to compare differences in latent cognitive variables including reward and punishment learning and choice consistency between groups.

RESULTS

AUDP and HC did not significantly differ with regard to direct perseveration rates after reversals. However, AUDP made overall less correct responses and specifically showed decreased win-stay behavior compared to HC. Interestingly, AUDP showed premature switching after no or little negative feedback but elevated proneness to stay when accumulation of negative feedback would make switching a more optimal option. Computational modeling revealed that AUDP compared to HC showed enhanced learning from punishment, a tendency to learn less from positive feedback and lower choice consistency.

CONCLUSION

Our data do not support the assumption that AUDP are characterized by increased perseveration behavior. Instead our findings provide evidence that enhanced negative reinforcement and decreased non-drug-related reward learning as well as diminished choice consistency underlie dysfunctional choice behavior in AUDP.

Cognitive Rigidity, Habitual Tendencies, and Obsessive-Compulsive Symptoms: Individual Differences and Compensatory Interactions

Recent theories have posited a range of cognitive risk factors for obsessive-compulsive disorder (OCD), including cognitive inflexibility and a maladaptive reliance on habits. However, empirical and methodological inconsistencies have obscured the understanding of whether inflexibility and habitual tendencies indeed shape OCD symptoms in clinical and sub-clinical populations, and whether there are notable interactions amongst these traits. The present investigation adopted an interactionist individual differences approach to examine the associations between behaviorally-assessed cognitive flexibility and subclinical OCD symptomatology in a healthy population. It also explored the nature of the interactions between cognitive flexibility and habitual tendencies, and the degree to which these cognitive traits predict subclinical OCD symptomatology. Across two studies, including a preregistration, Bayesian and regression analyses revealed that cognitive inflexibility and compulsive habitual tendencies act as unique and independent predictors of subclinical OCD symptomatology in healthy populations. Furthermore, there was a significant interaction between cognitive rigidity and habitual compulsivity, which accounted for 49.4% of the variance in subclinical OCD symptomatology in Study 1, and 37.3% in Study 2. In-depth analyses revealed a compensatory effect between cognitive inflexibility and habitual compulsivity such that both are necessary for OCD symptomatology, but neither is sufficient. These results imply that in order to generate reliable and nuanced models of the endophenotype of OCD symptomatology, it is essential to account for interactions between psychological traits. Moreover, the present findings have important implications for theories on the cognitive roots of OCD, and potentially in the development of interventions that target both cognitive inflexibility and habitual compulsivity.

Impaired Learning From Negative Feedback in Stimulant Use Disorder: Dopaminergic Modulation

BACKGROUND

Drug-induced alterations to the dopamine system in stimulant use disorder (SUD) are hypothesized to impair reinforcement learning (RL). Computational modeling enables the investigation of the latent processes of RL in SUD patients, which could elucidate the nature of their impairments.

METHODS

We investigated RL in 44 SUD patients and 41 healthy control participants using a probabilistic RL task that assesses learning from reward and punishment separately. In an independent sample, we determined the modulatory role of dopamine in RL following a single dose of the dopamine D2/3 receptor antagonist amisulpride (400 mg) and the agonist pramipexole (0.5 mg) in a randomised, double-blind, placebo-controlled, crossover design. We analyzed task performance using computational modelling and hypothesized that RL impairments in SUD patients would be differentially modulated by a dopamine D2/3 receptor antagonist and agonist.

RESULTS

Computational analyses in both samples revealed significantly reduced learning rates from punishment in SUD patients compared with healthy controls, whilst their reward learning rates were not measurably impaired. In addition, the dopaminergic receptor agents modulated RL parameters differentially in both groups. Both amisulpride and pramipexole impaired RL parameters in healthy participants, but ameliorated learning from punishment in SUD patients.

CONCLUSION

Our findings suggest that RL impairments seen in SUD patients are associated with altered dopamine function.

Association of Environmental Uncertainty With Altered Decision-making and Learning Mechanisms in Youths With Obsessive-Compulsive Disorder

Importance

Adults with obsessive-compulsive disorder (OCD) display perseverative behavior in stable environments but exhibit vacillating choice when payoffs are uncertain. These findings may be associated with intolerance of uncertainty and compulsive behaviors; however, little is known about the mechanisms underlying learning and decision-making in youths with OCD because research into this population has been limited.

Objective

To investigate cognitive mechanisms associated with decision-making in youths with OCD by using executive functioning tasks and computational modeling.

Design, Setting, and Participants

In this cross-sectional study, 50 youths with OCD (patients) and 53 healthy participants (controls) completed a probabilistic reversal learning (PRL) task between January 2014 and March 2020. A separate sample of 27 patients and 46 controls completed the Wisconsin Card Sorting Task (WCST) between January 2018 and November 2020. The study took place at the University of Cambridge in the UK.

Main Outcomes and Measures

Decision-making mechanisms were studied by fitting hierarchical bayesian reinforcement learning models to the 2 data sets and comparing model parameters between participant groups. Model parameters included reward and punishment learning rates (feedback sensitivity), reinforcement sensitivity and decision consistency (exploitation), and stickiness (perseveration). Associations of receipt of serotonergic medication with performance were assessed.

Results

In total, 50 patients (29 female patients [58%]; median age, 16.6 years [IQR, 15.3-18.0 years]) and 53 controls (30 female participants [57%]; median age, 16.4 years [IQR, 14.8-18.0 years]) completed the PRL task. A total of 27 patients (18 female patients [67%]; median age, 16.1 years [IQR, 15.2-17.2 years]) and 46 controls (28 female participants [61%]; median age, 17.2 [IQR, 16.3-17.6 years]) completed the WCST. During the reversal phase of the PRL task, patients made fewer correct responses (mean [SD] proportion: 0.83 [0.16] for controls and 0.61 [0.31] for patients; 95% CI, -1.31 to -0.64) and switched choices more often following false-negative feedback (mean [SD] proportion: 0.09 [0.16] for controls vs 0.27 [0.34] for patients; 95% CI, 0.60-1.26) and true-positive feedback (mean [SD] proportion: 0.93 [0.17] for controls vs 0.73 [0.34] for patients; 95% CI, -2.17 to -1.31). Computational modeling revealed that patients displayed enhanced reward learning rates (mean difference [MD], 0.21; 95% highest density interval [HDI], 0.04-0.38) but decreased punishment learning rates (MD, -0.29; 95% HDI, -0.39 to -0.18), reinforcement sensitivity (MD, -4.91; 95% HDI, -9.38 to -1.12), and stickiness (MD, -0.35; 95% HDI, -0.57 to -0.11) compared with controls. There were no group differences on standard WCST measures and computational model parameters. However, patients who received serotonergic medication showed slower response times (mean [SD], 1420.49 [279.71] milliseconds for controls, 1471.42 [212.81] milliseconds for patients who were unmedicated, and 1738.25 [349.23] milliseconds for patients who were medicated) (control vs medicated MD, -320.26 [95% CI, -547.00 to -88.68]) and increased unique errors (mean [SD] proportion: 0.001 [0.004] for controls, 0.002 [0.004] for patients who were unmedicated, and 0.008 [0.01] for patients who were medicated) (control vs medicated MD, -0.007 [95% CI, -3.14 to -0.36]) on the WCST.

Conclusions and Relevance

The results of this cross-sectional study indicated that youths with OCD showed atypical probabilistic reversal learning but were generally unimpaired on the deterministic WCST, although unexpected results were observed for patients receiving serotonergic medication. These findings have implications for reframing the understanding of early-onset OCD as a disorder in which decision-making is associated with uncertainty in the environment, a potential target for therapeutic treatment. These results provide continuity with findings in adults with OCD.

Examining the effects of psychoactive drugs on complex behavioral processes in laboratory animals

Behavioral pharmacology has been aided significantly by the development of innovative cognitive tasks designed to examine complex behavioral processes in laboratory animals. Performance outcomes under these conditions have provided key metrics of drug action which serve to supplement traditional in vivo assays of physiologic and behavioral effects of psychoactive drugs. This chapter provides a primer of cognitive tasks designed to assay different aspects of complex behavior, including learning, cognitive flexibility, memory, attention, motivation, and impulsivity. Both capstone studies and recent publications are highlighted throughout to illustrate task value for two distinct but often interconnected translational strategies. First, task performance in laboratory animals can be utilized to elucidate how drugs of abuse affect complex behavioral processes. Here, the expectation is that adverse effects on such processes will have predictive relevance to consequences that will be experienced by humans. Second, these same task outcomes can be used to evaluate candidate therapeutics. In this case, the extent to which drug doses with medicinal value perturb task performance can contribute critical information for a more complete safety profile appraisal and advance the process of medications development. Methodological and theoretical considerations are discussed and include an emphasis on determining selectivity in drug action on complex behavioral processes.

An Experimental Medicine Investigation of the Effects of Subacute Pramipexole Treatment on Emotional Information Processing in Healthy Volunteers

Treatment with the dopamine D2/D3 receptor agonist pramipexole has demonstrated promising clinical effects in patients with depression. However, the mechanisms through which pramipexole might alleviate depressive symptoms are currently not well understood. Conventional antidepressant drugs are thought to work by biasing the processing of emotional information in favour of positive relative to negative appraisal. In this study, we used an established experimental medicine assay to explore whether pramipexole treatment might have a similar effect. Employing a double-blind, parallel-group design, 40 healthy volunteers (aged 18 to 43 years, 50% female) were randomly allocated to 12 to 15 days of treatment with either pramipexole (at a peak daily dose of 1.0 mg pramipexole salt) or placebo. After treatment was established, emotional information processing was assessed on the neural level by measuring amygdala activity in response to positive and negative facial emotional expressions, using functional magnetic resonance imaging (MRI). In addition, behavioural measures of emotional information processing were collected at baseline and on drug, using an established computerized task battery, tapping into different cognitive domains. As predicted, pramipexole-treated participants, compared to those receiving placebo, showed decreased neural activity in response to negative (fearful) vs. positive (happy) facial expressions in bilateral amygdala. Contrary to our predictions, however, pramipexole treatment had no significant antidepressant-like effect on behavioural measures of emotional processing. This study provides the first experimental evidence that subacute pramipexole treatment in healthy volunteers modifies neural responses to emotional information in a manner that resembles the effects of conventional antidepressant drugs.

Are methamphetamine users compulsive? Faulty reinforcement learning, not inflexibility, underlies decision making in people with methamphetamine use disorder

Methamphetamine use disorder involves continued use of the drug despite negative consequences. Such 'compulsivity' can be measured by reversal learning tasks, which involve participants learning action-outcome task contingencies (acquisition-contingency) and then updating their behaviour when the contingencies change (reversal). Using these paradigms, animal models suggest that people with methamphetamine use disorder (PwMUD) may struggle to avoid repeating actions that were previously rewarded but are now punished (inflexibility). However, difficulties in learning task contingencies (reinforcement learning) may offer an alternative explanation, with meaningful treatment implications. We aimed to disentangle inflexibility and reinforcement learning deficits in 35 PwMUD and 32 controls with similar sociodemographic characteristics, using novel trial-by-trial analyses on a probabilistic reversal learning task. Inflexibility was defined as (a) weaker reversal phase performance, compared with the acquisition-contingency phases, and (b) persistence with the same choice despite repeated punishments. Conversely, reinforcement learning deficits were defined as (a) poor performance across both acquisition-contingency and reversal phases and (b) inconsistent postfeedback behaviour (i.e., switching after reward). Compared with controls, PwMUD exhibited weaker learning (odds ratio [OR] = 0.69, 95% confidence interval [CI] [0.63-0.77], p < .001), though no greater accuracy reduction during reversal. Furthermore, PwMUD were more likely to switch responses after one reward/punishment (OR = 0.83, 95% CI [0.77-0.89], p < .001; OR = 0.82, 95% CI [0.72-0.93], p = .002) but just as likely to switch after repeated punishments (OR = 1.03, 95% CI [0.73-1.45], p = .853). These results indicate that PwMUD's reversal learning deficits are driven by weaker reinforcement learning, not inflexibility.

The Role of Social Stress in the Development of Inhibitory Control Deficit: A Systematic Review in Preclinical Models

Inhibitory control deficit and impulsivity and compulsivity behaviours are present in different psychopathological disorders such as addiction, obsessive-compulsive disorders and schizophrenia, among others. Social relationships in humans and animals are governed by social organization rules, which modulate inhibitory control and coping strategies against stress. Social stress is associated with compulsive alcohol and drug use, pointing towards a determining factor in an increased vulnerability to inhibitory control deficit. The goal of the present review is to assess the implication of social stress and dominance on the vulnerability to develop impulsive and/or compulsive spectrum disorders, with the aid of the information provided by animal models. A systematic search strategy was carried out on the PubMed and Web of Science databases, and the most relevant information was structured in the text and tables. A total of 34 studies were recruited in the qualitative synthesis. The results show the role of social stress and dominance in increased drug and alcohol use, aggressive and impulsive behaviour. Moreover, the revised studies support the role of Dopaminergic (DA) activity and the alterations in the dopaminergic D1/D2 receptors as key factors in the development of inhibitory control deficit by social stress.

Frontoparietal hyperconnectivity during cognitive regulation in obsessive-compulsive disorder followed by reward valuation inflexibility

Obsessive-compulsive disorder (OCD) is characterized by cognitive deficits and altered reward processing systems. An imbalance between cognitive and reward pathways may explain the lack of control over obsessions followed by rewarding compulsive behaviors. While the processes of emotional cognitive regulation are widely studied in OCD, the mechanisms of cognitive regulation of reward are poorly described. Our goal was to investigate the OCD impact on cognitive regulation of reward at behavioral and neural functioning levels. OCD and control participants performed a functional magnetic resonance imaging task where they cognitively modulated their craving for food pictures under three cognitive regulation conditions: indulge/increase craving, distance/decrease craving, and natural/no regulation of craving. After regulation, the participants gave each picture a monetary value. We found that OCD patients had fixed food valuation scores while the control group modulated these values accordingly to the regulation conditions. Moreover, we observed frontoparietal hyperconnectivity during cognitive regulation. Our results suggest that OCD is characterized by deficits in cognitive regulation of internal states associated with inflexible behavior during reward processing. These findings bring new insights into the nature of compulsive behaviors in OCD.

Unlocking the reinforcement-learning circuits of the orbitofrontal cortex

Neuroimaging studies have consistently identified the orbitofrontal cortex (OFC) as being affected in individuals with neuropsychiatric disorders. OFC dysfunction has been proposed to be a key mechanism by which decision-making impairments emerge in diverse clinical populations, and recent studies employing computational approaches have revealed that distinct reinforcement-learning mechanisms of decision-making differ among diagnoses. In this perspective, we propose that these computational differences may be linked to select OFC circuits and present our recent work that has used a neurocomputational approach to understand the biobehavioral mechanisms of addiction pathology in rodent models. We describe how combining translationally analogous behavioral paradigms with reinforcement-learning algorithms and sophisticated neuroscience techniques in animals can provide critical insights into OFC pathology in biobehavioral disorders. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

A cross-species assessment of behavioral flexibility in compulsive disorders

Lack of behavioral flexibility has been proposed as one underlying cause of compulsions, defined as repetitive behaviors performed through rigid rituals. However, experimental evidence has proven inconsistent across human and animal models of compulsive-like behavior. In the present study, applying a similarly-designed reversal learning task in two different species, which share a common symptom of compulsivity (human OCD patients and Sapap3 KO mice), we found no consistent link between compulsive behaviors and lack of behavioral flexibility. However, we showed that a distinct subgroup of compulsive individuals of both species exhibit a behavioral flexibility deficit in reversal learning. This deficit was not due to perseverative, rigid behaviors as commonly hypothesized, but rather due to an increase in response lability. These cross-species results highlight the necessity to consider the heterogeneity of cognitive deficits in compulsive disorders and call for reconsidering the role of behavioral flexibility in the aetiology of compulsive behaviors.

Electroencephalographic Correlates of Obsessive-Compulsive Disorder

This chapter reviews EEG research in Obsessive-Compulsive Disorder (OCD), focusing on Event-Related Potentials (ERPs) such as the Contingent Negative Variation, N2, Error-Related Negativity, the feedback Error-Related Negativity and the Readiness Potential and their neural bases. The functional significance, utility and correlation of these ERPs with OCD symptoms will be discussed, alongside novel theories for integrating the research findings. I will consider hypotheses including goal-directed behaviour, overreliance on habits, dissociations between action and knowledge, and excessive intolerance of uncertainty in the context of EEG studies, thus providing a comprehensive framework of the electroencephalographic literature concerning OCD.

Computational Predictions for OCD Pathophysiology and Treatment: A Review

Obsessive compulsive disorder (OCD) can manifest as a debilitating disease with high degrees of co-morbidity as well as clinical and etiological heterogenity. However, the underlying pathophysiology is not clearly understood. Computational psychiatry is an emerging field in which behavior and its neural correlates are quantitatively analyzed and computational models are developed to improve understanding of disorders by comparing model predictions to observations. The aim is to more precisely understand psychiatric illnesses. Such computational and theoretical approaches may also enable more personalized treatments. Yet, these methodological approaches are not self-evident for clinicians with a traditional medical background. In this mini-review, we summarize a selection of computational OCD models and computational analysis frameworks, while also considering the model predictions from a perspective of possible personalized treatment. The reviewed computational approaches used dynamical systems frameworks or machine learning methods for modeling, analyzing and classifying patient data. Bayesian interpretations of probability for model selection were also included. The computational dissection of the underlying pathology is expected to narrow the explanatory gap between the phenomenological nosology and the neuropathophysiological background of this heterogeneous disorder. It may also contribute to develop biologically grounded and more informed dimensional taxonomies of psychopathology.