Decomposing the roles of perseveration and expected value representation in models of the Iowa gambling task

Test-retest reliability of the play-or-pass version of the Iowa Gambling Task

The Iowa Gambling Task (IGT) is used to assess decision-making in clinical populations. The original IGT does not disambiguate reward and punishment learning; however, an adaptation of the task, the "play-or-pass" IGT, was developed to better distinguish between reward and punishment learning. We evaluated the test-retest reliability of measures of reward and punishment learning from the play-or-pass IGT and examined associations with self-reported measures of reward/punishment sensitivity and internalizing symptoms. Participants completed the task across two sessions, and we calculated mean-level differences and rank-order stability of behavioral measures across the two sessions using traditional scoring, involving session-wide choice proportions, and computational modeling, involving estimates of different aspects of trial-level learning. Measures using both approaches were reliable; however, computational modeling provided more insights regarding between-session changes in performance, and how performance related to self-reported measures of reward/punishment sensitivity and internalizing symptoms. Our results show promise in using the play-or-pass IGT to assess decision-making; however, further work is still necessary to validate the play-or-pass IGT.

Modulation of dlPFC function and decision-making capacity by repetitive transcranial magnetic stimulation in methamphetamine use disorder

This study explores the impact of repetitive transcranial magnetic stimulation (rTMS) on decision-making capabilities in individuals with methamphetamine use disorder (MUD), alongside potential underlying psychological mechanisms. Employing the Iowa Gambling Task (IGT) and computational modeling techniques, we assessed the decision-making processes of 50 male MUD participants (24 underwent rTMS treatment, 26 received no treatment) and 39 healthy controls (HC). We compared pre- and post-rTMS treatment alterations in the left dorsolateral prefrontal cortex (dlPFC). Results revealed inferior performance in the IGT among the MUD group, characterized by aberrant model parameters in the Value-Plus-Perseverance (VPP) model, including heightened learning rate, outcome sensitivity, and reinforcement learning weight, alongside diminished response consistency and loss aversion. RTMS treatment demonstrated efficacy in reducing craving scores, enhancing decision-making abilities, and partially restoring normalcy to certain model parameters in the MUD cohort. Nonetheless, no linear relationship between changes in model parameters and craving was observed. These findings lend support to the somatic marker hypothesis, implicating the dlPFC in the decision-making deficits observed in MUD, with rTMS potentially ameliorating these deficits by modulating the function of these brain regions. This study not only offers novel insights and methodologies for MUD rehabilitation but also underscores the necessity for further research to corroborate and refine these findings. Trial Registration www.chictr.org.cn Identifier: No. ChiCTR17013610.

Decision-making deficits in obsessive-compulsive disorder are associated with abnormality of recency and response consistency parameter in prospect valence learning model

Background

Patients with obsessive-compulsive disorder (OCD) have deficits in decision-making in the Iowa Gambling Task (IGT). However, no study has investigated the parameters of the prospect valence learning (PVL) model in the IGT for OCD.

Aims

This study aimed to investigate deficits in decision-making in OCD using the PVL model and identify whether the parameters of the PVL model were associated with obsessive-compulsive severity.

Methods

Forty-seven medication-free patients with OCD were compared with 47 healthy controls (HCs). Decision-making was measured using the total net and block net scores of the IGT. A PVL model with a decay-reinforcement learning rule (PVL-DecayRI) was used to investigate the parameters of the model. Correlation analysis was conducted between each parameter of the PVL-DecayRL and obsessive-compulsive symptoms.

Results

The total net score of patients with OCD was significantly lower than that of the HCs. The block net scores of the OCD group did not differ across the five blocks, whereas in the HCs, the fifth block net score was significantly higher than the block net scores of the first and second blocks. The values of the recency and response consistency parameters of the PVL-DecayRI in patients with OCD were significantly lower than those in HCs. The recency parameter positively correlated with the Y-BOCS obsessive score. Meanwhile, there was no correlation between consistency parameter values and symptom severity in OCD.

Conclusion

Our detailed analysis of the decision-making deficit in OCD suggests that the most recent outcome has a small influence on the expectancy of prospect valence, as indicated by the lower recency parameter, and is characterized by more impulsive choices, as indicated by the lower consistency parameter.

Computational mechanisms underpinning greater exploratory behaviour in excess weight relative to healthy weight adolescents

Obesity in adolescence is associated with cognitive changes that lead to difficulties in shifting unhealthy habits in favour of alternative healthy behaviours, similar to addictive behaviours. An outstanding question is whether this shift in goal-directed behaviour is driven by over-exploitation or over-exploration of rewarding outcomes. Here, we addressed this question by comparing explore/exploit behaviour on the Iowa Gambling Task in 43 adolescents with excess weight against 38 adolescents with healthy weight. We computationally modelled both exploitation behaviour (e.g., reinforcement sensitivity and inverse decay parameters), and explorative behaviour (e.g., maximum directed exploration value). We found that overall, adolescents with excess weight displayed more behavioural exploration than their healthy-weight counterparts - specifically, demonstrating greater overall switching behaviour. Computational models revealed that this behaviour was driven by a higher maximum directed exploration value in the excess-weight group (U = 520.00, p = .005, BF₁₀ = 5.11). Importantly, however, we found substantial evidence that groups did not differ in reinforcement sensitivity (U = 867.00, p = .641, BF₁₀ = 0.30). Overall, our study demonstrates a preference for exploratory behaviour in adolescents with excess weight, independent of sensitivity to reward. This pattern could potentially underpin an intrinsic desire to explore energy-dense unhealthy foods - an as-yet untapped mechanism that could be targeted in future treatments of obesity in adolescents.

The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors

The obesogenic food environment includes easy access to highly-palatable, energy-dense, "ultra-processed" foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption-the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.

The outcome-representation learning model: impairments in decision-making in adolescents with excess weight

Impairments in decision-making have been suggested as a predisposing factor to obesity development. Individuals with excess weight display riskier decisions than normal weight people. Furthermore, adolescence is a period of life in which risky behavior may increase. We aimed to investigate decision making applying the Outcome-Representation-Learning (ORL) model to the Iowa Gambling Task (IGT) in adolescents with excess weight. Twenty-nine excess weight and twenty-eight normal weight adolescents, classified according to their age-adjusted body mass index (BMI) percentile, participated in the study. Decision-making was measured using the IGT. A Bayesian computational ORL model was applied to assess reward learning, punishment learning, forgetfulness, win perseverance and deck perseverance. The IGT net score was lower in excess weight than normal weight adolescents (β = 2.85; p < .027). Reward learning (95% HDI [0.011, 0.232]) was higher, while forgetfulness (95% HDI [− 0.711, − 0.181]) and deck perseverance (95% HDI [− 3.349, − 0.203]) were lower, in excess weight than normal weight adolescents. Excess weight adolescents seemed better at learning the most rewarding choices and showed a random strategy based on reward and novelty seeking. Consequently, excess weight adolescents made more disadvantageous selections, and performed worse in the IGT.

Computational models of exploration and exploitation characterise onset and efficacy of treatment in methamphetamine use disorder

People with Methamphetamine Use Disorder (PwMUD) spend substantial time and resources on substance use, which hinders their ability to explore alternate reinforcers. Gold‐standard behavioural treatments attempt to remedy this by encouraging action towards non‐drug reinforcers, but substance use often persists. We aimed to unravel the mechanistic drivers of this behaviour by applying a computational model of explore/exploit behaviour to decision‐making data (Iowa Gambling Task) from 106 PwMUD and 48 controls. We then examined the longitudinal link between explore/exploit mechanisms and changes in methamphetamine use 6 weeks later. Exploitation parameters included reinforcement sensitivity and inverse decay (i.e., number of past outcomes used to guide choices). Exploration parameters included maximum directed exploration value (i.e., value of trying novel actions). The Timeline Follow Back measured changes in methamphetamine use. Compared to controls, PwMUD showed deficits in exploitative decision‐making, characterised by reduced reinforcement sensitivity, U = 3065, p = 0.009, and less use of previous choice outcomes, U = 3062, p = 0.010. This was accompanied by a behavioural pattern of frequent shifting between choices, which appeared consistent with random exploration. Furthermore, PwMUD with greater reductions of methamphetamine use at 6 weeks had increased directed exploration (β = 0.22, p = 0.045); greater use of past choice outcomes (β = −0.39, p = 0.002) and greater choice consistency (β = −0.39, p = 0.002). Therefore, limited computational exploitation and increased behavioural exploration characterise PwMUD's presentation to treatment, while increased directed exploration, use of past choice outcomes and choice consistency predict greater reductions of methamphetamine use.

Computational basis of decision-making impairment in multiple sclerosis

BACKGROUND

Multiple sclerosis (MS) is commonly associated with decision-making, neurocognitive impairments, and mood and motivational symptoms. However, their relationship may be obscured by traditional scoring methods.

OBJECTIVES

To study the computational basis underlying decision-making impairments in MS and their interaction with neurocognitive and neuropsychiatric measures.

METHODS

Twenty-nine MS patients and 26 matched control subjects completed a computer version of the Iowa Gambling Task (IGT). Participants underwent neurocognitive evaluation using an expanded version of the Brief Repeatable Battery. Hierarchical Bayesian Analysis was used to estimate three established computational models to compare parameters between groups.

RESULTS

Patients showed increased learning rate and reduced loss-aversion during decision-making relative to control subjects. These alterations were associated with: (1) reduced net gains in the IGT; (2) processing speed, executive functioning and memory impairments; and (3) higher levels of depression and current apathy.

CONCLUSION

Decision-making deficits in MS patients could be described by the interplay between latent computational processes, neurocognitive impairments, and mood/motivational symptoms.

Using Methods From Computational Decision-making to Predict Nonadherence to Fitness Goals: Protocol for an Observational Study

Background

Can methods from computational models of decision-making be used to build a predictive model to identify individuals most likely to be nonadherent to personal fitness goals? Such a model may have significant value in the global battle against obesity. Despite growing awareness of the impact of physical inactivity on human health, sedentary behavior is increasingly linked to premature death in the developed world. The annual impact of sedentary behavior is significant, causing an estimated 2 million deaths. From a global perspective, sedentary behavior is one of the 10 leading causes of mortality and morbidity. Annually, considerable funding and countless public health initiatives are applied to promote physical fitness, with little impact on sustained behavioral change. Predictive models developed from multimodal methodologies combining data from decision-making tasks with contextual insights and objective physical activity data could be used to identify those most likely to abandon their fitness goals. This has the potential to enable development of more targeted support to ensure that those who embark on fitness programs are successful.

Objective

The aim of this study is to determine whether it is possible to use decision-making tasks such as the Iowa Gambling Task to help determine those most likely to abandon their fitness goals. Predictive models built using methods from computational models of decision-making, combining objective data from a fitness tracker with personality traits and modeling from decision-making games delivered via a mobile app, will be used to ascertain whether a predictive algorithm can identify digital personae most likely to be nonadherent to self-determined exercise goals. If it is possible to phenotype these individuals, it may be possible to tailor initiatives to support these individuals to continue exercising.

Methods

This is a siteless study design based on a bring your own device model. A total of 200 healthy adults who are novice exercisers and own a Fitbit (Fitbit Inc) physical activity tracker will be recruited via social media for this study. Participants will provide consent via the study app, which they will download from the Google Play store (Alphabet Inc) or Apple App Store (Apple Inc). They will also provide consent to share their Fitbit data. Necessary demographic information concerning age and sex will be collected as part of the recruitment process. Over 12 months, the scheduled study assessments will be pushed to the subjects to complete. The Iowa Gambling Task will be administered via a web app shared via a URL.

Results

Ethics approval was received from Dublin City University in December 2020. At manuscript submission, study recruitment was pending. The expected results will be published in 2022.

Conclusions

It is hoped that the study results will support the development of a predictive model and the study design will inform future research approaches.

Trial Registration

ClinicalTrials.gov NCT04783298; https://clinicaltrials.gov/ct2/show/NCT04783298

A Normative Account of Confirmation Bias During Reinforcement Learning

Reinforcement learning involves updating estimates of the value of states and actions on the basis of experience. Previous work has shown that in humans, reinforcement learning exhibits a confirmatory bias: when the value of a chosen option is being updated, estimates are revised more radically following positive than negative reward prediction errors, but the converse is observed when updating the unchosen option value estimate. Here, we simulate performance on a multi-arm bandit task to examine the consequences of a confirmatory bias for reward harvesting. We report a paradoxical finding: that confirmatory biases allow the agent to maximize reward relative to an unbiased updating rule. This principle holds over a wide range of experimental settings and is most influential when decisions are corrupted by noise. We show that this occurs because on average, confirmatory biases lead to overestimating the value of more valuable bandits and underestimating the value of less valuable bandits, rendering decisions overall more robust in the face of noise. Our results show how apparently suboptimal learning rules can in fact be reward maximizing if decisions are made with finite computational precision.

Feedback From Facial Expressions Contribute to Slow Learning Rate in an Iowa Gambling Task

Facial expressions of emotion can convey information about the world and disambiguate elements of the environment, thus providing direction to other people’s behavior. However, the functions of facial expressions from the perspective of learning patterns over time remain elusive. This study investigated how the feedback of facial expressions influences learning tasks in a context of ambiguity using the Iowa Gambling Task. The results revealed that the learning rate for facial expression feedback was slower in the middle of the learning period than it was for symbolic feedback. No difference was observed in deck selection or computational model parameters between the conditions, and no correlation was observed between task indicators and the results of depressive questionnaires.

Decision-Making Processes Related to Perseveration Are Indirectly Associated With Weight Status in Children Through Laboratory-Assessed Energy Intake

Decision-making contributes to what and how much we consume, and deficits in decision-making have been associated with increased weight status in children. Nevertheless, the relationships between cognitive and affective processes underlying decision-making (i.e., decision-making processes) and laboratory food intake are unclear. We used data from a four-session, within-subjects laboratory study to investigate the relationships between decision-making processes, food intake, and weight status in 70 children 7-to-11-years-old. Decision-making was assessed with the Hungry Donkey Task (HDT), a child-friendly task where children make selections with unknown reward outcomes. Food intake was measured with three paradigms: (1) a standard ad libitum meal, (2) an eating in the absence of hunger (EAH) protocol, and (3) a palatable buffet meal. Individual differences related to decision-making processes during the HDT were quantified with a reinforcement learning model. Path analyses were used to test whether decision-making processes that contribute to children's (a) expected value of a choice and (b) tendency to perseverate (i.e., repeatedly make the same choice) were indirectly associated with weight status through their effects on intake (kcal). Results revealed that increases in the tendency to perseverate after a gain outcome were positively associated with intake at all three paradigms and indirectly associated with higher weight status through intake at both the standard and buffet meals. Increases in the tendency to perseverate after a loss outcome were positively associated with EAH, but only in children whose tendency to perseverate persistedacross trials. Results suggest that decision-making processes that shape children's tendencies to repeat a behavior (i.e., perseverate) are related to laboratory energy intake across multiple eating paradigms. Children who are more likely to repeat a choice after a positive outcome have a tendency to eat more at laboratory meals. If this generalizes to contexts outside the laboratory, these children may be susceptible to obesity. By using a reinforcement learning model not previously applied to the study of eating behaviors, this study elucidated potential determinants of excess energy intake in children, which may be useful for the development of childhood obesity interventions.

Development of a novel computational model for the Balloon Analogue Risk Task: The Exponential-Weight Mean-Variance Model

The Balloon Analogue Risk Task (BART) is a popular task used to measure risk-taking behavior. To identify cognitive processes associated with choice behavior on the BART, a few computational models have been proposed. However, the extant models either fail to capture choice patterns on the BART or show poor parameter recovery performance. Here, we propose a novel computational model, the exponential-weight mean-variance (EWMV) model, which addresses the limitations of existing models. By using multiple model comparison methods, including post hoc model fits criterion and parameter recovery, we showed that the EWMV model outperforms the existing models. In addition, we applied the EWMV model to BART data from healthy controls and substance-using populations (patients with past opiate and stimulant dependence). The results suggest that (1) the EWMV model addresses the limitations of existing models and (2) heroin-dependent individuals show reduced risk preference than other groups, which may have significant clinical implications.

Risk-taking unmasked: Using risky choice and temporal discounting to explain COVID-19 preventative behaviors

To reduce the spread of COVID-19 transmission, government agencies in the United States (US) recommended precautionary guidelines, including wearing masks and social distancing to encourage the prevention of the disease. However, compliance with these guidelines has been inconsistent. This correlational study examined whether individual differences in risky decision-making and motivational propensities predicted compliance with COVID-19 preventative behaviors in a sample of US adults (N = 404). Participants completed an online study from September through December 2020 that included a risky choice decision-making task, temporal discounting task, and measures of appropriate mask-wearing, social distancing, and perceived risk of engaging in public activities. Linear regression results indicated that greater temporal discounting and risky decision-making were associated with less appropriate mask-wearing behavior and social distancing. Additionally, demographic factors, including personal experience with COVID-19 and financial difficulties due to COVID-19, were also associated with differences in COVID-19 preventative behaviors. Path analysis results showed that risky decision-making behavior, temporal discounting, and risk perception collectively predicted 55% of the variance in appropriate mask-wearing behavior. Individual differences in general decision-making patterns are therefore highly predictive of who complies with COVID-19 prevention guidelines.

The self’s choice: Priming attentional focus on bodily self promotes loss frequency bias

When attention is focused on self representation(s), the ability to evaluate one’s internal sensations is enhanced, according to previous research by Ainley and colleagues (Consciousness and Cognition, 22(4), 1231–1238, 2013). Self-representations are usually distinguished between bodily and narrative. Both bodily and narrative representations improve decision-making processes, in that the consideration of alternatives is informed by sensations experienced deep inside the body (e.g., anxiety) as suggest by the literature (Noël, Brevers & Bechara in Frontiers in Psychiatry, 4, 179,  2013). The objective of the present study is to analyze the decision-making process in multiple conditions of stimulated self-representations. Participants played the Iowa Gambling Task three times (a baseline without stimuli and two randomly ordered stimulations to prime bodily and narrative self-representations). While no significant differences emerged regarding advantageous choices, participants showed loss frequency bias in the condition with bodily-self representation priming. Two interpretations are proposed: bodily-self focus acted as a distractor diminishing participants’ commitment to long term outcomes or enhanced interoception promoted aversion to losses. Directions are given for future research and clinical implications.

Recollecting Cross-Cultural Evidences: Are Decision Makers Really Foresighted in Iowa Gambling Task?

The Iowa Gambling Task (IGT) has become a remarkable experimental paradigm of dynamic emotion decision making. In recent years, research has emphasized the "prominent deck B (PDB) phenomenon" among normal (control group) participants, in which they favor "bad" deck B with its high-frequency gain structure-a finding that is incongruent with the original IGT hypothesis concerning foresightedness. Some studies have attributed such performance inconsistencies to cultural differences. In the present review, 86 studies featuring data on individual deck selections were drawn from an initial sample of 958 IGT-related studies published from 1994 to 2017 for further investigation. The PDB phenomenon was found in 67.44% of the studies (58 of 86), and most participants were recorded as having adopted the "gain-stay loss-randomize" strategy to cope with uncertainty. Notably, participants in our sample of studies originated from 16 areas across North America, South America, Europe, Oceania, and Asia, and the findings suggest that the PDB phenomenon may be cross-cultural.

Autonomic responses to choice outcomes: Links to task performance and reinforcement-learning parameters

Previous work has shown that autonomic responses to choice feedback can predict subsequent decision-making. In this study, we tested whether skin conductance responses (SCRs) and heart rate (HR) decelerations following the presentation of choice outcomes predict Iowa Gambling Task performance in nonclinical participants (n = 64). We also examined how these signals related to parameters of a reinforcement-learning (RL) model. Feedback SCRs and HR decelerations were greater following outcomes that included losses and choices from the bad decks defined by their negative expected value. In addition, SCRs predicted task performance. A hierarchical Bayesian RL model indicated that greater feedback SCR for the bad decks compared to good decks was associated with stronger loss aversion and a lower learning rate, both of which predicted higher performance. These results suggest that feedback-related SCRs are linked to individual differences in outcome evaluation and learning processes that guide reinforcement-learning.

Decomposing conditioned avoidance performance with computational models

Avoidance towards innocuous stimuli is a key characteristic across anxiety-related disorders and chronic pain. Insights into the relevant learning processes of avoidance are often gained via laboratory procedures, where individuals learn to avoid stimuli or movements that have been previously associated with an aversive stimulus. Typically, statistical analyses of data gathered with conditioned avoidance procedures include frequency data, for example, the number of times a participant has avoided an aversive stimulus. Here, we argue that further insights into the underlying processes of avoidance behavior could be unraveled using computational models of behavior. We then demonstrate how computational models could be used by reanalysing a previously published avoidance data set and interpreting the key findings. We conclude our article by listing some challenges in the direct application of computational modeling to avoidance data sets.

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Involvement of dopamine in regulating exploration during decision-making has long been hypothesized, but direct causal evidence in humans is still lacking. Here, we use a combination of computational modeling, pharmacological intervention and functional magnetic resonance imaging to address this issue. Thirty-one healthy male participants performed a restless four-armed bandit task in a within-subjects design under three drug conditions: 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist haloperidol, and placebo. Choices were best explained by an extension of an established Bayesian learning model accounting for perseveration, directed exploration and random exploration. Modeling revealed attenuated directed exploration under L-dopa, while neural signatures of exploration, exploitation and prediction error were unaffected. Instead, L-dopa attenuated neural representations of overall uncertainty in insula and dorsal anterior cingulate cortex. Our results highlight the computational role of these regions in exploration and suggest that dopamine modulates how this circuit tracks accumulating uncertainty during decision-making.

Mutual benefits: Combining reinforcement learning with sequential sampling models

Reinforcement learning models of error-driven learning and sequential-sampling models of decision making have provided significant insight into the neural basis of a variety of cognitive processes. Until recently, model-based cognitive neuroscience research using both frameworks has evolved separately and independently. Recent efforts have illustrated the complementary nature of both modelling traditions and showed how they can be integrated into a unified theoretical framework, explaining trial-by-trial dependencies in choice behavior as well as response time distributions. Here, we review a theoretical background of integrating the two classes of models, and review recent empirical efforts towards this goal. We furthermore argue that the integration of both modelling traditions provides mutual benefits for both fields, and highlight promises of this approach for cognitive modelling and model-based cognitive neuroscience.

Sequential exploration in the Iowa gambling task: Validation of a new computational model in a large dataset of young and old healthy participants

The Iowa Gambling Task (IGT) is one of the most common paradigms used to assess decision-making and executive functioning in neurological and psychiatric disorders. Several reinforcement-learning (RL) models were recently proposed to refine the qualitative and quantitative inferences that can be made about these processes based on IGT data. Yet, these models do not account for the complex exploratory patterns which characterize participants' behavior in the task. Using a dataset of more than 500 subjects, we demonstrate the existence of sequential exploration in the IGT and we describe a new computational architecture disentangling exploitation, random exploration and sequential exploration in this large population of participants. The new Value plus Sequential Exploration (VSE) architecture provided a better fit than previous models. Parameter recovery, model recovery and simulation analyses confirmed the superiority of the VSE scheme. Furthermore, using the VSE model, we confirmed the existence of a significant reduction in directed exploration across lifespan in the IGT, as previously reported with other paradigms. Finally, we provide a user-friendly toolbox enabling researchers to easily and flexibly fit computational models on the IGT data, hence promoting reanalysis of the numerous datasets acquired in various populations of patients and contributing to the development of computational psychiatry.

Pure correlates of exploration and exploitation in the human brain

Balancing exploration and exploitation is a fundamental problem in reinforcement learning. Previous neuroimaging studies of the exploration-exploitation dilemma could not completely disentangle these two processes, making it difficult to unambiguously identify their neural signatures. We overcome this problem using a task in which subjects can either observe (pure exploration) or bet (pure exploitation). Insula and dorsal anterior cingulate cortex showed significantly greater activity on observe trials compared to bet trials, suggesting that these regions play a role in driving exploration. A model-based analysis of task performance suggested that subjects chose to observe until a critical evidence threshold was reached. We observed a neural signature of this evidence accumulation process in the ventromedial prefrontal cortex. These findings support theories positing an important role for anterior cingulate cortex in exploration, while also providing a new perspective on the roles of insula and ventromedial prefrontal cortex.

The relative merit of empirical priors in non-identifiable and sloppy models: Applications to models of learning and decision-making : Empirical priors

Formal modeling approaches to cognition provide a principled characterization of observed responses in terms of a set of postulated processes, specifically in terms of parameters that modulate the latter. These model-based characterizations are useful to the extent that there is a clear, one-to-one mapping between parameters and model expectations (identifiability) and that parameters can be recovered from reasonably sized data using a typical experimental design (recoverability). These properties are sometimes not met for certain combinations of model classes and data. One suggestion to improve parameter identifiability and recoverability involves the use of "empirical priors", which constrain parameters according to a previously observed distribution of values. We assessed the efficacy of this proposal using a combination of real and artificial data. Our results showed that a point-estimate variant of the empirical-prior method could not improve parameter recovery systematically. We identified the source of poor parameter recovery in the low information content of the data. As a follow-up step, we developed a fully Bayesian variant of the empirical-prior method and assessed its performance. We find that even such a method that takes the covariance structure of the parameter distributions into account cannot reliably improve parameter recovery. We conclude that researchers should invest additional efforts in improving the informativeness of their experimental designs, as many of the problems associated to impoverished designs cannot be alleviated by modern statistical methods alone.

Is the Clinical Version of the Iowa Gambling Task Relevant for Assessing Choice Behavior in Cases of Internet Addiction?

Objective: A critical issue in research related to the Iowa gambling task (IGT) is the use of the alternative factors expected value and gain-loss frequency to distinguish between clinical cases and control groups. When the IGT has been used to examine cases of Internet addiction (IA), the literature reveals inconsistencies in the results. However, few studies have utilized the clinical version of IGT (cIGT) to examine IA cases. The present study aims to resolve previous inconsistencies and to examine the validity of the cIGT by comparing performances of controls with cases of Internet gaming disorder (IGD), a subtype of IA defined by the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders. Methods: The study recruited 23 participants with clinically diagnosed IGD and 38 age-matched control participants. Based on the basic assumptions of IGT and the gain-loss frequency viewpoint, a dependent variables analysis was carried out. Results: The results showed no statistical difference between the two groups in most performance indices and therefore support the findings of most IGT-IA studies; in particular, expected value and gain-loss frequency did not distinguish between the IGD cases and controls. However, the participants in both groups were influenced by the gain-loss frequency, revealing the existence of the prominent deck B phenomenon. Conclusion: The findings provide two possible interpretations. The first is that choice behavior deficits do not constitute a characteristic feature of individuals who have been diagnosed with IGD/IA. The second is that, as the cIGT was unable to distinguish the choice behavior of the IGD/IA participants from that of controls, the cIGT may not be relevant for assessing IGD based on the indices provided by the expected value and gain-loss frequency perspectives in the standard administration of IGT.

The Outcome-Representation Learning Model: A Novel Reinforcement Learning Model of the Iowa Gambling Task

The Iowa Gambling Task (IGT) is widely used to study decision-making within healthy and psychiatric populations. However, the complexity of the IGT makes it difficult to attribute variation in performance to specific cognitive processes. Several cognitive models have been proposed for the IGT in an effort to address this problem, but currently no single model shows optimal performance for both short- and long-term prediction accuracy and parameter recovery. Here, we propose the Outcome-Representation Learning (ORL) model, a novel model that provides the best compromise between competing models. We test the performance of the ORL model on 393 subjects' data collected across multiple research sites, and we show that the ORL reveals distinct patterns of decision-making in substance-using populations. Our work highlights the importance of using multiple model comparison metrics to make valid inference with cognitive models and sheds light on learning mechanisms that play a role in underweighting of rare events.

Better late than never (or early): Music training in late childhood is associated with enhanced decision-making

Decision-making is critical to everyday life. Here we ask: to what extent does music training benefit decision-making? Supported by strong associations between music training and enhanced cross-domain skills, we hypothesize that musicians may show decision-making advantages relative to non-musicians. Prior work has also argued for a "critical period" for cross-domain plasticity such that beginning music training early enhances sensorimotor brain regions that mature early in life. Given that brain regions supporting decision-making begin maturing late in childhood, we hypothesized that an advantage in decision-making may only be present in musicians who began music training later in childhood. To test this hypothesis, young adults who began music training before and after 8 years of age (early-trained musicians, ET; late-trained musicians, LT, respectively) and non-musicians (NM) performed a decision-making task. We found a decision-making advantage in LT relative to ET and NM. To better understand the mechanism of the LT advantage, we conducted computational modeling on participant responses and found that LT were less biased by recent outcomes and incorporated longer strings of outcomes when deciding among the choice options. These results tentatively suggest that music training may confer decision-making enhancements, and carry strong implications for the utility of music training in childhood.

Age of onset of cannabis use and decision making under uncertainty

Objective

Cannabis, like other substances, negatively affects health, inducing respiratory problems and mental and cognitive alterations. Memory and learning disorders, as well as executive dysfunctions, are also neuropsychological disorders associated to cannabis use. Recent evidence reveals that cannabis use during adolescence may disrupt the normal development of the brain. This study is aimed to analyze possible differences between early-onset and late-onset cannabis consumers.

Method

We used a task based on a card game with four decks and different programs of gains/losses. A total of 72 subjects (19 women; 53 men) participated in the study; they were selected through a purposive sampling and divided into three groups: early-onset consumers, late-onset consumers, and control (non-consumers). The task used was the “Cartas” program (computerized version based on the Iowa Gambling Task (IGT)), with two versions: direct and inverse. The computational model “Prospect Valence Learning” (PVL) was applied in order to describe the decision according to four characteristics: utility, loss aversion, recency, and consistency.

Results

The results evidence worst performance in the IGT in the early-onset consumers as compared to late-onset consumers and control. Differences between groups were also found in the PVL computational model parameters, since the process of decision making of the early-onset consumers was more influenced by the magnitude of the gains-losses, and more determined by short-term results without loss aversion.

Conclusions

Early onset cannabis use may involve decision-making problems, and therefore intervention programs are necessary in order to reduce the prevalence and delay the onset of cannabis use among teenagers.

A Case of Divergent Predictions Made by Delta and Decay Rule Learning Models

The Delta and Decay rules are two learning rules used to update expected values in reinforcement learning (RL) models. The delta rule learns average rewards, whereas the decay rule learns cumulative rewards for each option. Participants learned to select between pairs of options that had reward probabilities of .65 (option A) versus .35 (option B) or .75 (option C) versus .25 (option D) on separate trials in a binary-outcome choice task. Crucially, during training there were twice as AB trials as CD trials, therefore participants experienced more cumulative reward from option A even though option C had a higher average reward rate (.75 versus .65). Participants then decided between novel combinations of options (e.g, A versus C). The Decay model predicted more A choices, but the Delta model predicted more C choices, because those respective options had higher cumulative versus average reward values. Results were more in line with the Decay model's predictions. This suggests that people may retrieve memories of cumulative reward to compute expected value instead of learning average rewards for each option.

Reinforcement learning models of risky choice and the promotion of risk-taking by losses disguised as wins in rats

Risky decisions are inherently characterized by the potential to receive gains or incur losses, and these outcomes have distinct effects on subsequent decision-making. One important factor is that individuals engage in loss-chasing, in which the reception of a loss is followed by relatively increased risk-taking. Unfortunately, the mechanisms of loss-chasing are poorly understood, despite the potential importance for understanding pathological choice behavior. The goal of the present experiment was to illuminate the mechanisms governing individual differences in loss-chasing and risky-choice behaviors. Rats chose between a low-uncertainty outcome that always delivered a variable amount of reward and a high-uncertainty outcome that probabilistically delivered reward. Loss-processing and loss-chasing were assessed in the context of losses disguised as wins (LDWs), which are loss outcomes that are presented along with gain-related stimuli. LDWs have been suggested to interfere with adaptive decision-making in humans and thus potentially increase loss-making. Here, the rats presented with LDWs were riskier, in that they made more choices for the high-uncertainty outcome. A series of nonlinear models were fit to individual rats' data to elucidate the possible psychological mechanisms that best account for individual differences in high-uncertainty choices and loss-chasing behaviors. The models suggested that the rats presented with LDWs were more prone to showing a stay bias following high-uncertainty outcomes compared to rats not presented with LDWs. These results collectively suggest that LDWs acquire conditioned reinforcement properties that encourage continued risk-taking and increase loss-chasing following previous high-risk decisions. (PsycINFO Database Record

Bayesian techniques for analyzing group differences in the Iowa Gambling Task: A case study of intuitive and deliberate decision-makers

The Iowa Gambling Task (IGT) is one of the most popular experimental paradigms for comparing complex decision-making across groups. Most commonly, IGT behavior is analyzed using frequentist tests to compare performance across groups, and to compare inferred parameters of cognitive models developed for the IGT. Here, we present a Bayesian alternative based on Bayesian repeated-measures ANOVA for comparing performance, and a suite of three complementary model-based methods for assessing the cognitive processes underlying IGT performance. The three model-based methods involve Bayesian hierarchical parameter estimation, Bayes factor model comparison, and Bayesian latent-mixture modeling. We illustrate these Bayesian methods by applying them to test the extent to which differences in intuitive versus deliberate decision style are associated with differences in IGT performance. The results show that intuitive and deliberate decision-makers behave similarly on the IGT, and the modeling analyses consistently suggest that both groups of decision-makers rely on similar cognitive processes. Our results challenge the notion that individual differences in intuitive and deliberate decision styles have a broad impact on decision-making. They also highlight the advantages of Bayesian methods, especially their ability to quantify evidence in favor of the null hypothesis, and that they allow model-based analyses to incorporate hierarchical and latent-mixture structures.

How the threat of losses makes people explore more than the promise of gains

Until recently, loss aversion has been inferred exclusively from choice asymmetries in the loss and gain domains. This study examines the impact of the prospect of losses on exploratory search in a situation in which exploration is costly. Taking advantage of the largest available data set of decisions from experience, analyses showed that most people explore payoff distributions more under the threat of a loss than under the promise of a gain. This behavioral regularity thus occurs in both costly search and cost-free search (see Lejarraga, Hertwig, & Gonzalez, Cognition, 124, 334–342, 2012). Furthermore, a model comparison identified the simple win-stay-lose-shift heuristic as a likely candidate mechanism behind the loss–gain exploration asymmetry observed. In contrast, models assuming loss aversion failed to reproduce the asymmetry. Moreover, the asymmetry was not simply a precursor of loss aversion but a phenomenon separate from it. These findings are consistent with the recently proposed notion of intensified vigilance in the face of potential losses.

A tutorial on bridge sampling

The marginal likelihood plays an important role in many areas of Bayesian statistics such as parameter estimation, model comparison, and model averaging. In most applications, however, the marginal likelihood is not analytically tractable and must be approximated using numerical methods. Here we provide a tutorial on bridge sampling (Bennett, 1976; Meng & Wong, 1996), a reliable and relatively straightforward sampling method that allows researchers to obtain the marginal likelihood for models of varying complexity. First, we introduce bridge sampling and three related sampling methods using the beta-binomial model as a running example. We then apply bridge sampling to estimate the marginal likelihood for the Expectancy Valence (EV) model-a popular model for reinforcement learning. Our results indicate that bridge sampling provides accurate estimates for both a single participant and a hierarchical version of the EV model. We conclude that bridge sampling is an attractive method for mathematical psychologists who typically aim to approximate the marginal likelihood for a limited set of possibly high-dimensional models.

Shared and Disorder-Specific Neurocomputational Mechanisms of Decision-Making in Autism Spectrum Disorder and Obsessive-Compulsive Disorder

Autism spectrum disorder (ASD) and obsessive-compulsive disorder (OCD) often share phenotypes of repetitive behaviors, possibly underpinned by abnormal decision-making. To compare neural correlates underlying decision-making between these disorders, brain activation of boys with ASD (N = 24), OCD (N = 20) and typically developing controls (N = 20) during gambling was compared, and computational modeling compared performance. Patients were unimpaired on number of risky decisions, but modeling showed that both patient groups had lower choice consistency and relied less on reinforcement learning compared to controls. ASD individuals had disorder-specific choice perseverance abnormalities compared to OCD individuals. Neurofunctionally, ASD and OCD boys shared dorsolateral/inferior frontal underactivation compared to controls during decision-making. During outcome anticipation, patients shared underactivation compared to controls in lateral inferior/orbitofrontal cortex and ventral striatum. During reward receipt, ASD boys had disorder-specific enhanced activation in inferior frontal/insular regions relative to OCD boys and controls. Results showed that ASD and OCD individuals shared decision-making strategies that differed from controls to achieve comparable performance to controls. Patients showed shared abnormalities in lateral-(orbito)fronto-striatal reward circuitry, but ASD boys had disorder-specific lateral inferior frontal/insular overactivation, suggesting that shared and disorder-specific mechanisms underpin decision-making in these disorders. Findings provide evidence for shared neurobiological substrates that could serve as possible future biomarkers.

Task complexity moderates the influence of descriptions in decisions from experience

Decisions-makers often have access to a combination of descriptive and experiential information, but limited research so far has explored decisions made using both. Three experiments explore the relationship between task complexity and the influence of descriptions. We show that in simple experience-based decision-making tasks, providing congruent descriptions has little influence on task performance in comparison to experience alone without descriptions, since learning via experience is relatively easy. In more complex tasks, which are slower and more demanding to learn experientially, descriptions have stronger influence and help participants identify their preferred choices. However, when the task gets too complex to be concisely described, the influence of descriptions is reduced hence showing a non-monotonic pattern of influence of descriptions according to task complexity. We also propose a cognitive model that incorporates descriptive information into the traditional reinforcement learning framework, with the impact of descriptions moderated by task complexity. This model fits the observed behavior better than previous models and replicates the observed non-monotonic relationship between impact of descriptions and task complexity. This research has implications for the development of effective warning labels that rely on simple descriptive information to trigger safer behavior in complex environments.

Revealing Neurocomputational Mechanisms of Reinforcement Learning and Decision-Making With the hBayesDM Package

Reinforcement learning and decision-making (RLDM) provide a quantitative framework and computational theories with which we can disentangle psychiatric conditions into the basic dimensions of neurocognitive functioning. RLDM offer a novel approach to assessing and potentially diagnosing psychiatric patients, and there is growing enthusiasm for both RLDM and computational psychiatry among clinical researchers. Such a framework can also provide insights into the brain substrates of particular RLDM processes, as exemplified by model-based analysis of data from functional magnetic resonance imaging (fMRI) or electroencephalography (EEG). However, researchers often find the approach too technical and have difficulty adopting it for their research. Thus, a critical need remains to develop a user-friendly tool for the wide dissemination of computational psychiatric methods. We introduce an R package called hBayesDM (hierarchical Bayesian modeling of Decision-Making tasks), which offers computational modeling of an array of RLDM tasks and social exchange games. The hBayesDM package offers state-of-the-art hierarchical Bayesian modeling, in which both individual and group parameters (i.e., posterior distributions) are estimated simultaneously in a mutually constraining fashion. At the same time, the package is extremely user-friendly: users can perform computational modeling, output visualization, and Bayesian model comparisons, each with a single line of coding. Users can also extract the trial-by-trial latent variables (e.g., prediction errors) required for model-based fMRI/EEG. With the hBayesDM package, we anticipate that anyone with minimal knowledge of programming can take advantage of cutting-edge computational-modeling approaches to investigate the underlying processes of and interactions between multiple decision-making (e.g., goal-directed, habitual, and Pavlovian) systems. In this way, we expect that the hBayesDM package will contribute to the dissemination of advanced modeling approaches and enable a wide range of researchers to easily perform computational psychiatric research within different populations.

Neural substrates of updating the prediction through prediction error during decision making

Learning of prediction error (PE), including reward PE and risk PE, is crucial for updating the prediction in reinforcement learning (RL). Neurobiological and computational models of RL have reported extensive brain activations related to PE. However, the occurrence of PE does not necessarily predict updating the prediction, e.g., in a probability-known event. Therefore, the brain regions specifically engaged in updating the prediction remain unknown. Here, we conducted two functional magnetic resonance imaging (fMRI) experiments, the probability-unknown Iowa Gambling Task (IGT) and the probability-known risk decision task (RDT). Behavioral analyses confirmed that PEs occurred in both tasks but were only used for updating the prediction in the IGT. By comparing PE-related brain activations between the two tasks, we found that the rostral anterior cingulate cortex/ventral medial prefrontal cortex (rACC/vmPFC) and the posterior cingulate cortex (PCC) activated only during the IGT and were related to both reward and risk PE. Moreover, the responses in the rACC/vmPFC and the PCC were modulated by uncertainty and were associated with reward prediction-related brain regions. Electric brain stimulation over these regions lowered the performance in the IGT but not in the RDT. Our findings of a distributed neural circuit of PE processing suggest that the rACC/vmPFC and the PCC play a key role in updating the prediction through PE processing during decision making.

Choice in experiential learning: True preferences or experimental artifacts?

The rate of selecting different options in the decisions-from-feedback paradigm is commonly used to measure preferences resulting from experiential learning. While convergence to a single option increases with experience, some variance in choice remains even when options are static and offer fixed rewards. Employing a decisions-from-feedback paradigm followed by a policy-setting task, we examined whether the observed variance in choice is driven by factors related to the paradigm itself: Continued exploration (e.g., believing options are non-stationary) or exploitation of perceived outcome patterns (i.e., a belief that sequential choices are not independent). Across two studies, participants showed variance in their choices, which was related (i.e., proportional) to the policies they set. In addition, in Study 2, participants' reported under-confidence was associated with the amount of choice variance in later choices and policies. These results suggest that variance in choice is better explained by participants lacking confidence in knowing which option is better, rather than methodological artifacts (i.e., exploration or failures to recognize outcome independence). As such, the current studies provide evidence for the decisions-from-feedback paradigm's validity as a behavioral research method for assessing learned preferences.

Dopamine, depressive symptoms, and decision-making: the relationship between spontaneous eye blink rate and depressive symptoms predicts Iowa Gambling Task performance

Depressive symptomatology has been associated with alterations in decision-making, although conclusions have been mixed, with depressed individuals showing impairments in some contexts but advantages in others. The dopaminergic system may link depressive symptoms with decision-making performance. We assessed the role of striatal dopamine D2 receptor density, using spontaneous eye blink rates, in moderating the relationship between depressive symptoms and decision-making performance in a large undergraduate sample that had not been screened for mental illness (N = 104). The regression results revealed that eye blink rate moderated the relationship between depressive symptoms and advantageous decisions on the Iowa Gambling Task, in which individuals with more depressive symptomatology and high blink rates (higher striatal dopamine D2 receptor density) performed better on the task. Our computational modeling results demonstrated that depressive symptoms alone were associated with enhanced loss-aversive behavior, whereas individuals with high blink rates and elevated depressive symptoms tended to persevere in selecting options that led to net gains (avoiding options with net losses). These findings suggest that variation in striatal dopamine D2 receptor availability in individuals with depressive symptoms may contribute to differences in decision-making behavior.

A Simplified Model of Choice Behavior under Uncertainty

The Iowa Gambling Task (IGT) has been standardized as a clinical assessment tool (Bechara, 2007). Nonetheless, numerous research groups have attempted to modify IGT models to optimize parameters for predicting the choice behavior of normal controls and patients. A decade ago, most researchers considered the expected utility (EU) model (Busemeyer and Stout, 2002) to be the optimal model for predicting choice behavior under uncertainty. However, in recent years, studies have demonstrated that models with the prospect utility (PU) function are more effective than the EU models in the IGT (Ahn et al., 2008). Nevertheless, after some preliminary tests based on our behavioral dataset and modeling, it was determined that the Ahn et al. (2008) PU model is not optimal due to some incompatible results. This study aims to modify the Ahn et al. (2008) PU model to a simplified model and used the IGT performance of 145 subjects as the benchmark data for comparison. In our simplified PU model, the best goodness-of-fit was found mostly as the value of α approached zero. More specifically, we retested the key parameters α, λ, and A in the PU model. Notably, the influence of the parameters α, λ, and A has a hierarchical power structure in terms of manipulating the goodness-of-fit in the PU model. Additionally, we found that the parameters λ and A may be ineffective when the parameter α is close to zero in the PU model. The present simplified model demonstrated that decision makers mostly adopted the strategy of gain-stay loss-shift rather than foreseeing the long-term outcome. However, there are other behavioral variables that are not well revealed under these dynamic-uncertainty situations. Therefore, the optimal behavioral models may not have been found yet. In short, the best model for predicting choice behavior under dynamic-uncertainty situations should be further evaluated.

Neural correlates of state-based decision-making in younger and older adults

Older and younger adults performed a state-based decision-making task while undergoing functional MRI (fMRI). We proposed that younger adults would be more prone to base their decisions on expected value comparisons, but that older adults would be more reactive decision-makers who would act in response to recent changes in rewards or states, rather than on a comparison of expected values. To test this we regressed BOLD activation on two measures from a sophisticated reinforcement learning (RL) model. A value-based regressor was computed by subtracting the immediate value of the selected alternative from its long-term value. The other regressor was a state-change uncertainty signal that served as a proxy for whether the participant's state improved or declined, relative to the previous trial. Younger adults' activation was modulated by the value-based regressor in ventral striatal and medial PFC regions implicated in reinforcement learning. Older adults' activation was modulated by state-change uncertainty signals in right dorsolateral PFC, and activation in this region was associated with improved performance in the task. This suggests that older adults may depart from standard expected-value based strategies and recruit lateral PFC regions to engage in reactive decision-making strategies.

An improved cognitive model of the Iowa and Soochow Gambling Tasks with regard to model fitting performance and tests of parameter consistency

The Iowa Gambling Task (IGT) and the Soochow Gambling Task (SGT) are two experience-based risky decision-making tasks for examining decision-making deficits in clinical populations. Several cognitive models, including the expectancy-valence learning (EVL) model and the prospect valence learning (PVL) model, have been developed to disentangle the motivational, cognitive, and response processes underlying the explicit choices in these tasks. The purpose of the current study was to develop an improved model that can fit empirical data better than the EVL and PVL models and, in addition, produce more consistent parameter estimates across the IGT and SGT. Twenty-six opiate users (mean age 34.23; SD 8.79) and 27 control participants (mean age 35; SD 10.44) completed both tasks. Eighteen cognitive models varying in evaluation, updating, and choice rules were fit to individual data and their performances were compared to that of a statistical baseline model to find a best fitting model. The results showed that the model combining the prospect utility function treating gains and losses separately, the decay-reinforcement updating rule, and the trial-independent choice rule performed the best in both tasks. Furthermore, the winning model produced more consistent individual parameter estimates across the two tasks than any of the other models.

Older Adults are Highly Responsive to Recent Events During Decision-Making

Recent work suggests that older adults' decision-making behavior is highly affected by recent events. In the present work younger and older adults performed a two-choice task where one option provided a larger average reward, but there was a large amount of noise around the mean reward for each option which led to sharp improvements or declines in rewards over trials. Older adults showed greater responsiveness to recent events than younger adults as evidenced by fits of Reinforcement Learning (RL) models. Older adults were particularly sensitive to recent negative events, which was evidenced by a strong tendency for older adults to switch to the other option following steep declines in reward. This tendency led to superior performance for older adults in one condition where heightened sensitivity to recent negative events was advantageous. These results extend prior work that has found an older adult bias toward negative feedback, and suggest that older adults engage in more abrupt switching in response to negative outcomes than younger adults.

Decision-making in stimulant and opiate addicts in protracted abstinence: evidence from computational modeling with pure users

Substance dependent individuals (SDI) often exhibit decision-making deficits; however, it remains unclear whether the nature of the underlying decision-making processes is the same in users of different classes of drugs and whether these deficits persist after discontinuation of drug use. We used computational modeling to address these questions in a unique sample of relatively “pure” amphetamine-dependent (N = 38) and heroin-dependent individuals (N = 43) who were currently in protracted abstinence, and in 48 healthy controls (HC). A Bayesian model comparison technique, a simulation method, and parameter recovery tests were used to compare three cognitive models: (1) Prospect Valence Learning with decay reinforcement learning rule (PVL-DecayRI), (2) PVL with delta learning rule (PVL-Delta), and (3) Value-Plus-Perseverance (VPP) model based on Win-Stay-Lose-Switch (WSLS) strategy. The model comparison results indicated that the VPP model, a hybrid model of reinforcement learning (RL) and a heuristic strategy of perseverance had the best post-hoc model fit, but the two PVL models showed better simulation and parameter recovery performance. Computational modeling results suggested that overall all three groups relied more on RL than on a WSLS strategy. Heroin users displayed reduced loss aversion relative to HC across all three models, which suggests that their decision-making deficits are longstanding (or pre-existing) and may be driven by reduced sensitivity to loss. In contrast, amphetamine users showed comparable cognitive functions to HC with the VPP model, whereas the second best-fitting model with relatively good simulation performance (PVL-DecayRI) revealed increased reward sensitivity relative to HC. These results suggest that some decision-making deficits persist in protracted abstinence and may be mediated by different mechanisms in opiate and stimulant users.

Effects of emotion on prospection during decision-making

In two experiments we examined the role of emotion, specifically worry, anxiety, and mood, on prospection during decision-making. Worry is a particularly relevant emotion to study in the context of prospection because high levels of worry may make individuals more aversive toward the uncertainty associated with the prospect of obtaining future improvements in rewards or states. Thus, high levels of worry might lead to reduced prospection during decision-making and enhance preference for immediate over delayed rewards. In Experiment 1 participants performed a two-choice dynamic decision-making task where they were required to choose between one option (the decreasing option) which provided larger immediate rewards but declines in future states, and another option (the increasing option) which provided smaller immediate rewards but improvements in future states, making it the optimal choice. High levels of worry were associated with poorer performance in the task. Additionally, fits of a sophisticated reinforcement-learning model that incorporated both reward-based and state-based information suggested that individuals reporting high levels of worry gave greater weight to the immediate rewards they would receive on each trial than to the degree to which each action would lead to improvements in their future state. In Experiment 2 we found that high levels of worry were associated with greater delay discounting using a standard delay discounting task. Combined, the results suggest that high levels of worry are associated with reduced prospection during decision-making. We attribute these results to high worriers' aversion toward the greater uncertainty associated with attempting to improve future rewards than to maximize immediate reward. These results have implications for researchers interested in the effects of emotion on cognition, and suggest that emotion strongly affects the focus on temporal outcomes during decision-making.

A Comparison Model of Reinforcement-Learning and Win-Stay-Lose-Shift Decision-Making Processes: A Tribute to W.K. Estes

W.K. Estes often championed an approach to model development whereby an existing model was augmented by the addition of one or more free parameters, and a comparison between the simple and more complex, augmented model determined whether the additions were justified. Following this same approach we utilized Estes' (1950) own augmented learning equations to improve the fit and plausibility of a win-stay-lose-shift (WSLS) model that we have used in much of our recent work. Estes also championed models that assumed a comparison between multiple concurrent cognitive processes. In line with this, we develop a WSLS-Reinforcement Learning (RL) model that assumes that the output of a WSLS process that provides a probability of staying or switching to a different option based on the last two decision outcomes is compared with the output of an RL process that determines a probability of selecting each option based on a comparison of the expected value of each option. Fits to data from three different decision-making experiments suggest that the augmentations to the WSLS and RL models lead to a better account of decision-making behavior. Our results also support the assertion that human participants weigh the output of WSLS and RL processes during decision-making.