Impulsivity and predictive control are associated with suboptimal action-selection and action-value learning in regular gamblers

Computational approaches to modeling gambling behaviour: Opportunities for understanding disordered gambling

Computational modeling has become an important tool in neuroscience and psychiatry research to provide insight into the cognitive processes underlying normal and pathological behavior. There are two modeling frameworks, reinforcement learning (RL) and drift diffusion modeling (DDM), that are well-developed in cognitive science, and have begun to be applied to Gambling Disorder. RL models focus on explaining how an agent uses reward to learn about the environment and make decisions based on outcomes. The DDM is a binary choice framework that breaks down decision making into psychologically meaningful components based on choice reaction time analyses. Both approaches have begun to yield insight into aspects of cognition that are important for, but not unique to, gambling, and thus relevant to the development of Gambling Disorder. However, these approaches also oversimplify or neglect various aspects of decision making seen in real-world gambling behavior. Gambling Disorder presents an opportunity for 'bespoke' modeling approaches to consider these neglected components. In this review, we discuss studies that have used RL and DDM frameworks to investigate some of the key cognitive components in gambling and Gambling Disorder. We also include an overview of Bayesian models, a methodology that could be useful for more tailored modeling approaches. We highlight areas in which computational modeling could enable progression in the investigation of the cognitive mechanisms relevant to gambling.

Impulsivity Relates to Multi-Trial Choice Strategy in Probabilistic Reversal Learning

Impulsivity is defined as a trait-like tendency to engage in rash actions that are poorly thought out or expressed in an untimely manner. Previous research has found that impulsivity relates to deficits in decision making, in particular when it necessitates executive control or reward outcomes. Reinforcement learning (RL) relies on the ability to integrate reward or punishment outcomes to make good decisions, and has recently been shown to often recruit executive function; as such, it is unsurprising that impulsivity has been studied in the context of RL. However, how impulsivity relates to the mechanisms of RL remains unclear. We aimed to investigate the relationship between impulsivity and learning in a reward-driven learning task with probabilistic feedback and reversal known to recruit executive function. Based on prior literature in clinical populations, we predicted that higher impulsivity would be associated with poorer performance on the task, driven by more frequent switching following unrewarded outcomes. Our results did not support this prediction, but more advanced, trial-history dependent analyses revealed specific effects of impulsivity on switching behavior following consecutive unrewarded trials. Computational modeling captured group-level behavior, but not impulsivity results. Our results support previous findings highlighting the importance of sensitivity to negative outcomes in understanding how impulsivity relates to learning, but indicate that this may stem from more complex strategies than usually considered in computational models of learning. This should be an important target for future research.

Attenuated Directed Exploration during Reinforcement Learning in Gambling Disorder

Gambling disorder (GD) is a behavioral addiction associated with impairments in value-based decision-making and behavioral flexibility and might be linked to changes in the dopamine system. Maximizing long-term rewards requires a flexible trade-off between the exploitation of known options and the exploration of novel options for information gain. This exploration-exploitation trade-off is thought to depend on dopamine neurotransmission. We hypothesized that human gamblers would show a reduction in directed (uncertainty-based) exploration, accompanied by changes in brain activity in a fronto-parietal exploration-related network. Twenty-three frequent, non-treatment seeking gamblers and twenty-three healthy matched controls (all male) performed a four-armed bandit task during functional magnetic resonance imaging (fMRI). Computational modeling using hierarchical Bayesian parameter estimation revealed signatures of directed exploration, random exploration, and perseveration in both groups. Gamblers showed a reduction in directed exploration, whereas random exploration and perseveration were similar between groups. Neuroimaging revealed no evidence for group differences in neural representations of basic task variables (expected value, prediction errors). Our hypothesis of reduced frontal pole (FP) recruitment in gamblers was not supported. Exploratory analyses showed that during directed exploration, gamblers showed reduced parietal cortex and substantia-nigra/ventral-tegmental-area activity. Cross-validated classification analyses revealed that connectivity in an exploration-related network was predictive of group status, suggesting that connectivity patterns might be more predictive of problem gambling than univariate effects. Findings reveal specific reductions of strategic exploration in gamblers that might be linked to altered processing in a fronto-parietal network and/or changes in dopamine neurotransmission implicated in GD.SIGNIFICANCE STATEMENT Wiehler et al. (2021) report that gamblers rely less on the strategic exploration of unknown, but potentially better rewards during reward learning. This is reflected in a related network of brain activity. Parameters of this network can be used to predict the presence of problem gambling behavior in participants.

Decision-making (in)flexibility in gambling disorder

BACKGROUND

Behavioral flexibility -the ability to dynamically readjust our behavior in response to reward contingency changes- is often investigated using probabilistic reversal learning tasks (PRLT). Poor PRLT performance has been proposed as a proxy for compulsivity, and theorized to be related to perseverative gambling. Previous attempts to measure inflexibility with the PRLT in patients with gambling disorder have, however, used a variety of indices that may conflate inflexibility with more general aspects of performance in the task.

METHODS

Trial-by-trial PRLT acquisition and reacquisition curves in 84 treatment-seeking patients with gambling disorder and 64 controls (non-gamblers and non-problem recreational gamblers) were analyzed to distinguish between (a) variability in acquisition learning, and (b) reacquisition learning in reversed contingency phases. Complementarily, stay/switch responses throughout the task were analyzed to identify (c) premature switching, and (d) sensitivity to accumulated negative feedback.

RESULTS AND INTERPRETATION

Even after controlling for differences in acquisition learning, patients were slower to readjust their behavior in reversed contingency phases, and were more prone to maintain their decisions despite accumulated negative feedback. Inflexibility in patients with gambling disorder is thus a robust phenomenon that could predate gambling escalation, or result from massive exposure to gambling activities.

Gambling disorder is associated with reduced sensitivity to expected value during risky choice

BACKGROUND AND AIMS

Individuals with gambling disorder display increased levels of risk-taking, but it is not known if it is associated with an altered subjective valuation of gains and/or losses, perception of their probabilities, or integration of these sources of information into expected value.

METHODS

Participants with gambling disorder (n = 48) were compared with a healthy comparison group (n = 35) on a two-choice lottery task that involved either gains-only or losses-only gambles. On each trial, two lotteries were displayed, showing the associated probability and magnitude of the possible outcome for each. On each trial, participants chose one of the two lotteries, and the outcome was revealed.

RESULTS

Choice behaviour was highly sensitive to the expected value of the two gambles in both the gain and loss domains. This sensitivity to expected value was attenuated in the group with gambling disorder. The group with gambling disorder used both probability and magnitude information less, and this impairment was greater for probability information. By contrast, they used prior feedback (win vs loss) to inform their next choice, despite the independence of each trial. Within the gambling disorder group, problem gambling severity and trait gambling-related cognitions independently predicted reduced sensitivity to expected value. The majority of observed effects were consistent across both gain and loss domains.

DISCUSSION AND CONCLUSIONS

Our results provide a thorough characterization of decision processes in gain and loss domains in gambling disorder, and place these problems in the context of theoretical constructs from behavioural economics.

The Neurobiology of Impulsive Decision-Making and Reinforcement Learning in Nonhuman Animals

Impulsive decisions are those that favor immediate over delayed rewards, involve the acceptance of undue risk or uncertainty, or fail to adapt to environmental changes. Pathological levels of impulsive decision-making have been observed in individuals with mental illness, but there may be substantial heterogeneity in the processes that drive impulsive choices. Understanding this behavioral heterogeneity may be critical for understanding associated diverseness in the neural mechanisms that give rise to impulsivity. The application of reinforcement learning algorithms in the deconstruction of impulsive decision-making phenotypes can help bridge the gap between biology and behavior and provide insights into the biobehavioral heterogeneity of impulsive choice. This chapter will review the literature on the neurobiological mechanisms of impulsive decision-making in nonhuman animals; specifically, the role of the amine neuromodulatory systems (dopamine, serotonin, norepinephrine, and acetylcholine) in impulsive decision-making and reinforcement learning processes is discussed. Ultimately, the integration of reinforcement learning algorithms with sophisticated behavioral and neuroscience techniques may be critical for advancing the understanding of the neurochemical basis of impulsive decision-making.

Increased motor impulsivity in a rat gambling task during chronic ropinirole treatment: potentiation by win-paired audiovisual cues

RATIONALE

Chronic administration of D_2/3 receptor agonists ropinirole or pramipexole can increase the choice of uncertain rewards in rats, theoretically approximating iatrogenic gambling disorder (iGD).

OBJECTIVES

We aimed to assess the effect of chronic ropinirole in animal models that attempt to capture critical aspects of commercial gambling, including the risk of losing rather than failing to gain, and the use of win-paired sensory stimuli heavily featured in electronic gambling machines (EGMs).

METHODS

Male Long-Evans rats learned the rat gambling task (rGT; n = 24), in which animals sample between four options that differ in the magnitude and probability of rewards and time-out punishments. In the cued rGT (n = 40), reward-concurrent audiovisual cues were added that scaled in complexity with win size. Rats were then implanted with an osmotic pump delivering ropinirole (5 mg/kg/day) or saline for 28 days.

RESULTS

Chronic ropinirole did not unequivocally increase preference for more uncertain outcomes in either the cued or uncued rGT. Ropinirole transiently increased premature responses, a measure of motor impulsivity, and this change was larger and more long-lasting in the cued task.

CONCLUSIONS

These data suggest that explicitly signaling loss prevents the increase in preference for uncertain rewards caused by D_2/3 receptor agonists observed previously. The ability of win-paired cues to amplify ropinirole-induced increases in motor impulsivity may explain why compulsive use of EGMs is particularly common in iGD. These data offer valuable insight into the cognitive-behavioral mechanisms through which chronic dopamine agonist treatments may induce iGD and related impulse control disorders.