An improved cognitive model of the Iowa and Soochow Gambling Tasks with regard to model fitting performance and tests of parameter consistency

Decision-making deficits in obsessive-compulsive disorder are associated with abnormality of recency and response consistency parameter in prospect valence learning model

Background

Patients with obsessive-compulsive disorder (OCD) have deficits in decision-making in the Iowa Gambling Task (IGT). However, no study has investigated the parameters of the prospect valence learning (PVL) model in the IGT for OCD.

Aims

This study aimed to investigate deficits in decision-making in OCD using the PVL model and identify whether the parameters of the PVL model were associated with obsessive-compulsive severity.

Methods

Forty-seven medication-free patients with OCD were compared with 47 healthy controls (HCs). Decision-making was measured using the total net and block net scores of the IGT. A PVL model with a decay-reinforcement learning rule (PVL-DecayRI) was used to investigate the parameters of the model. Correlation analysis was conducted between each parameter of the PVL-DecayRL and obsessive-compulsive symptoms.

Results

The total net score of patients with OCD was significantly lower than that of the HCs. The block net scores of the OCD group did not differ across the five blocks, whereas in the HCs, the fifth block net score was significantly higher than the block net scores of the first and second blocks. The values of the recency and response consistency parameters of the PVL-DecayRI in patients with OCD were significantly lower than those in HCs. The recency parameter positively correlated with the Y-BOCS obsessive score. Meanwhile, there was no correlation between consistency parameter values and symptom severity in OCD.

Conclusion

Our detailed analysis of the decision-making deficit in OCD suggests that the most recent outcome has a small influence on the expectancy of prospect valence, as indicated by the lower recency parameter, and is characterized by more impulsive choices, as indicated by the lower consistency parameter.

Using Methods From Computational Decision-making to Predict Nonadherence to Fitness Goals: Protocol for an Observational Study

Background

Can methods from computational models of decision-making be used to build a predictive model to identify individuals most likely to be nonadherent to personal fitness goals? Such a model may have significant value in the global battle against obesity. Despite growing awareness of the impact of physical inactivity on human health, sedentary behavior is increasingly linked to premature death in the developed world. The annual impact of sedentary behavior is significant, causing an estimated 2 million deaths. From a global perspective, sedentary behavior is one of the 10 leading causes of mortality and morbidity. Annually, considerable funding and countless public health initiatives are applied to promote physical fitness, with little impact on sustained behavioral change. Predictive models developed from multimodal methodologies combining data from decision-making tasks with contextual insights and objective physical activity data could be used to identify those most likely to abandon their fitness goals. This has the potential to enable development of more targeted support to ensure that those who embark on fitness programs are successful.

Objective

The aim of this study is to determine whether it is possible to use decision-making tasks such as the Iowa Gambling Task to help determine those most likely to abandon their fitness goals. Predictive models built using methods from computational models of decision-making, combining objective data from a fitness tracker with personality traits and modeling from decision-making games delivered via a mobile app, will be used to ascertain whether a predictive algorithm can identify digital personae most likely to be nonadherent to self-determined exercise goals. If it is possible to phenotype these individuals, it may be possible to tailor initiatives to support these individuals to continue exercising.

Methods

This is a siteless study design based on a bring your own device model. A total of 200 healthy adults who are novice exercisers and own a Fitbit (Fitbit Inc) physical activity tracker will be recruited via social media for this study. Participants will provide consent via the study app, which they will download from the Google Play store (Alphabet Inc) or Apple App Store (Apple Inc). They will also provide consent to share their Fitbit data. Necessary demographic information concerning age and sex will be collected as part of the recruitment process. Over 12 months, the scheduled study assessments will be pushed to the subjects to complete. The Iowa Gambling Task will be administered via a web app shared via a URL.

Results

Ethics approval was received from Dublin City University in December 2020. At manuscript submission, study recruitment was pending. The expected results will be published in 2022.

Conclusions

It is hoped that the study results will support the development of a predictive model and the study design will inform future research approaches.

Trial Registration

ClinicalTrials.gov NCT04783298; https://clinicaltrials.gov/ct2/show/NCT04783298

Neural regions associated with gain-loss frequency and average reward in older and younger adults

Research on the biological basis of reinforcement-learning has focused on how brain regions track expected value based on average reward. However, recent work suggests that humans are more attuned to reward frequency. Furthermore, older adults are less likely to use expected values to guide choice than younger adults. This raises the question of whether brain regions assumed to be sensitive to average reward, like the medial and lateral PFC, also track reward frequency, and whether there are age-based differences. Older (60-81 years) and younger (18-30 years) adults performed the Soochow Gambling task, which separates reward frequency from average reward, while undergoing fMRI. Overall, participants preferred options that provided negative net payoffs, but frequent gains. Older adults improved less over time, were more reactive to recent negative outcomes, and showed greater frequency-related activation in several regions, including DLPFC. We also found broader recruitment of prefrontal and parietal regions associated with frequency value and reward prediction errors in older adults, which may indicate compensation. The results suggest greater reliance on average reward for younger adults than older adults.

Component Processes of Decision Making in a Community Sample of Precariously Housed Persons: Associations With Learning and Memory, and Health-Risk Behaviors

The Iowa Gambling Task (IGT) is a widely used measure of decision making, but its value in signifying behaviors associated with adverse, "real-world" consequences has not been consistently demonstrated in persons who are precariously housed or homeless. Studies evaluating the ecological validity of the IGT have primarily relied on traditional IGT scores. However, computational modeling derives underlying component processes of the IGT, which capture specific facets of decision making that may be more closely related to engagement in behaviors associated with negative consequences. This study employed the Prospect Valence Learning (PVL) model to decompose IGT performance into component processes in 294 precariously housed community residents with substance use disorders. Results revealed a predominant focus on gains and a lack of sensitivity to losses in these vulnerable community residents. Hypothesized associations were not detected between component processes and self-reported health-risk behaviors. These findings provide insight into the processes underlying decision making in a vulnerable substance-using population and highlight the challenge of linking specific decision making processes to "real-world" behaviors.

The Performance of College Students on the Iowa Gambling Task: Differences Between Scoring Approaches

The Iowa Gambling Task (IGT) is one of the most common behavioral decision-making tasks used in clinical and research settings. Less-than-expected performance among healthy adults generates concerns about the validity of this task, and it is possible the particular scoring approach utilized could impact interpretation. We examined how performance patterns changed across several scoring approaches, utilizing a large, college student sample, both with (n = 406) and without (n = 1,547) a self-reported history of psychiatric or other diagnosis. Higher net scores were seen when participants selected decks with a low loss frequency than decks with high long-term outcomes; however, participants overall underperformed the IGT normative data sample. Receiver operating characteristic curves examining multiple scoring approaches revealed no threshold of impaired performance that both maximized sensitivity and minimized false positive rate on the IGT. Scoring approach matters in the determination of impaired decision making via the IGT in adults.

Understanding Human Decision Making in an Interactive Landslide Simulator Tool via Reinforcement Learning

Prior research has used an Interactive Landslide Simulator (ILS) tool to investigate human decision making against landslide risks. It has been found that repeated feedback in the ILS tool about damages due to landslides causes an improvement in human decisions against landslide risks. However, little is known on how theories of learning from feedback (e.g., reinforcement learning) would account for human decisions in the ILS tool. The primary goal of this paper is to account for human decisions in the ILS tool via computational models based upon reinforcement learning and to explore the model mechanisms involved when people make decisions in the ILS tool. Four different reinforcement-learning models were developed and evaluated in their ability to capture human decisions in an experiment involving two conditions in the ILS tool. The parameters of an Expectancy-Valence (EV) model, two Prospect-Valence-Learning models (PVL and PVL-2), a combination EV-PU model, and a random model were calibrated to human decisions in the ILS tool across the two conditions. Later, different models with their calibrated parameters were generalized to data collected in an experiment involving a new condition in ILS. When generalized to this new condition, the PVL-2 model’s parameters of both damage-feedback conditions outperformed all other RL models (including the random model). We highlight the implications of our results for decision making against landslide risks.

Cognition and addiction

In this targeted review, we summarize current knowledge on substance-use disorder (SUD)-related cognitive deficits, the link between these deficits and clinical outcomes, and the cognitive training, remediation, and pharmacological approaches that have the potential to rescue cognition. We conclude that: (i) people with SUDs have moderate deficits in memory, attention, executive functions, and decision-making (including reward expectancy, valuation, and learning); (ii) deficits in higher-order executive functions and decision-making are significant predictors of relapse; (iii) cognitive training programs targeting reward-related appetitive biases, cognitive remediation strategies targeting goal-based decision-making, and pharmacotherapies targeting memory, attention, and impulsivity have potential to rescue SUD-related cognitive deficits. We suggest avenues for future research, including developing brief, clinically oriented harmonized cognitive testing suites to improve individualized prediction of treatment outcomes; computational modeling that can achieve deep phenotyping of cognitive subtypes likely to respond to different interventions; and phenotype-targeted cognitive, pharmacological, and combined interventions. We conclude with a tentative model of neuroscience-informed precision medicine.


Effortful goal-directed behavior in schizophrenia: Computational subtypes and associations with cognition

Schizophrenia is associated with amotivation and reduced goal-directed behavior, which have been linked to poor functional outcomes. Motivational deficits in schizophrenia are often measured using effort-based decision-making (EBDM) paradigms, revealing consistent alterations in effort expenditure relative to controls. Although these results have generally been interpreted in terms of decreased motivation, the ability to use trial-by-trial changes in reward magnitude or probability of receipt to guide effort allocation may also be affected by cognitive deficits. To date, it remains unclear whether altered performance in EBDM primarily reflects deficits in motivation, cognitive functioning, or both. We applied a newly developed computational modeling approach to the analysis of EBDM data from two previously collected samples comprising 153 patients and 105 controls to determine the extent to which individuals did or did not use available information about reward and probability to guide effort allocation. Half of the participants with schizophrenia failed to incorporate information about reward and probability when making effort-expenditure decisions. The subset of patients who exhibited difficulties using reward and probability information were characterized by greater impairments across measures of cognitive functioning. Interestingly, even within the subset of patients who successfully used reward and probability information to guide effort expenditure, higher levels of negative symptoms related to motivation and avolition were associated with greater effort aversion during the task. Taken together, these data suggest that prior reports of aberrant EBDM in schizophrenia patients are related to both cognitive function and individual differences in negative symptoms. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Is the Clinical Version of the Iowa Gambling Task Relevant for Assessing Choice Behavior in Cases of Internet Addiction?

Objective: A critical issue in research related to the Iowa gambling task (IGT) is the use of the alternative factors expected value and gain-loss frequency to distinguish between clinical cases and control groups. When the IGT has been used to examine cases of Internet addiction (IA), the literature reveals inconsistencies in the results. However, few studies have utilized the clinical version of IGT (cIGT) to examine IA cases. The present study aims to resolve previous inconsistencies and to examine the validity of the cIGT by comparing performances of controls with cases of Internet gaming disorder (IGD), a subtype of IA defined by the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders. Methods: The study recruited 23 participants with clinically diagnosed IGD and 38 age-matched control participants. Based on the basic assumptions of IGT and the gain-loss frequency viewpoint, a dependent variables analysis was carried out. Results: The results showed no statistical difference between the two groups in most performance indices and therefore support the findings of most IGT-IA studies; in particular, expected value and gain-loss frequency did not distinguish between the IGD cases and controls. However, the participants in both groups were influenced by the gain-loss frequency, revealing the existence of the prominent deck B phenomenon. Conclusion: The findings provide two possible interpretations. The first is that choice behavior deficits do not constitute a characteristic feature of individuals who have been diagnosed with IGD/IA. The second is that, as the cIGT was unable to distinguish the choice behavior of the IGD/IA participants from that of controls, the cIGT may not be relevant for assessing IGD based on the indices provided by the expected value and gain-loss frequency perspectives in the standard administration of IGT.

Bayesian techniques for analyzing group differences in the Iowa Gambling Task: A case study of intuitive and deliberate decision-makers

The Iowa Gambling Task (IGT) is one of the most popular experimental paradigms for comparing complex decision-making across groups. Most commonly, IGT behavior is analyzed using frequentist tests to compare performance across groups, and to compare inferred parameters of cognitive models developed for the IGT. Here, we present a Bayesian alternative based on Bayesian repeated-measures ANOVA for comparing performance, and a suite of three complementary model-based methods for assessing the cognitive processes underlying IGT performance. The three model-based methods involve Bayesian hierarchical parameter estimation, Bayes factor model comparison, and Bayesian latent-mixture modeling. We illustrate these Bayesian methods by applying them to test the extent to which differences in intuitive versus deliberate decision style are associated with differences in IGT performance. The results show that intuitive and deliberate decision-makers behave similarly on the IGT, and the modeling analyses consistently suggest that both groups of decision-makers rely on similar cognitive processes. Our results challenge the notion that individual differences in intuitive and deliberate decision styles have a broad impact on decision-making. They also highlight the advantages of Bayesian methods, especially their ability to quantify evidence in favor of the null hypothesis, and that they allow model-based analyses to incorporate hierarchical and latent-mixture structures.

A Neural Circuit Mechanism for the Involvements of Dopamine in Effort-Related Choices: Decay of Learned Values, Secondary Effects of Depletion, and Calculation of Temporal Difference Error

Dopamine has been suggested to be crucially involved in effort-related choices. Key findings are that dopamine depletion (i) changed preference for a high-cost, large-reward option to a low-cost, small-reward option, (ii) but not when the large-reward option was also low-cost or the small-reward option gave no reward, (iii) while increasing the latency in all the cases but only transiently, and (iv) that antagonism of either dopamine D1 or D2 receptors also specifically impaired selection of the high-cost, large-reward option. The underlying neural circuit mechanisms remain unclear. Here we show that findings i–iii can be explained by the dopaminergic representation of temporal-difference reward-prediction error (TD-RPE), whose mechanisms have now become clarified, if (1) the synaptic strengths storing the values of actions mildly decay in time and (2) the obtained-reward-representing excitatory input to dopamine neurons increases after dopamine depletion. The former is potentially caused by background neural activity–induced weak synaptic plasticity, and the latter is assumed to occur through post-depletion increase of neural activity in the pedunculopontine nucleus, where neurons representing obtained reward exist and presumably send excitatory projections to dopamine neurons. We further show that finding iv, which is nontrivial given the suggested distinct functions of the D1 and D2 corticostriatal pathways, can also be explained if we additionally assume a proposed mechanism of TD-RPE calculation, in which the D1 and D2 pathways encode the values of actions with a temporal difference. These results suggest a possible circuit mechanism for the involvements of dopamine in effort-related choices and, simultaneously, provide implications for the mechanisms of TD-RPE calculation.

A tutorial on bridge sampling

The marginal likelihood plays an important role in many areas of Bayesian statistics such as parameter estimation, model comparison, and model averaging. In most applications, however, the marginal likelihood is not analytically tractable and must be approximated using numerical methods. Here we provide a tutorial on bridge sampling (Bennett, 1976; Meng & Wong, 1996), a reliable and relatively straightforward sampling method that allows researchers to obtain the marginal likelihood for models of varying complexity. First, we introduce bridge sampling and three related sampling methods using the beta-binomial model as a running example. We then apply bridge sampling to estimate the marginal likelihood for the Expectancy Valence (EV) model-a popular model for reinforcement learning. Our results indicate that bridge sampling provides accurate estimates for both a single participant and a hierarchical version of the EV model. We conclude that bridge sampling is an attractive method for mathematical psychologists who typically aim to approximate the marginal likelihood for a limited set of possibly high-dimensional models.

Exploration and recency as the main proximate causes of probability matching: a reinforcement learning analysis

Research has not yet reached a consensus on why humans match probabilities instead of maximise in a probability learning task. The most influential explanation is that they search for patterns in the random sequence of outcomes. Other explanations, such as expectation matching, are plausible, but do not consider how reinforcement learning shapes people's choices. We aimed to quantify how human performance in a probability learning task is affected by pattern search and reinforcement learning. We collected behavioural data from 84 young adult participants who performed a probability learning task wherein the majority outcome was rewarded with 0.7 probability, and analysed the data using a reinforcement learning model that searches for patterns. Model simulations indicated that pattern search, exploration, recency (discounting early experiences), and forgetting may impair performance. Our analysis estimated that 85% (95% HDI [76, 94]) of participants searched for patterns and believed that each trial outcome depended on one or two previous ones. The estimated impact of pattern search on performance was, however, only 6%, while those of exploration and recency were 19% and 13% respectively. This suggests that probability matching is caused by uncertainty about how outcomes are generated, which leads to pattern search, exploration, and recency.

Task complexity moderates the influence of descriptions in decisions from experience

Decisions-makers often have access to a combination of descriptive and experiential information, but limited research so far has explored decisions made using both. Three experiments explore the relationship between task complexity and the influence of descriptions. We show that in simple experience-based decision-making tasks, providing congruent descriptions has little influence on task performance in comparison to experience alone without descriptions, since learning via experience is relatively easy. In more complex tasks, which are slower and more demanding to learn experientially, descriptions have stronger influence and help participants identify their preferred choices. However, when the task gets too complex to be concisely described, the influence of descriptions is reduced hence showing a non-monotonic pattern of influence of descriptions according to task complexity. We also propose a cognitive model that incorporates descriptive information into the traditional reinforcement learning framework, with the impact of descriptions moderated by task complexity. This model fits the observed behavior better than previous models and replicates the observed non-monotonic relationship between impact of descriptions and task complexity. This research has implications for the development of effective warning labels that rely on simple descriptive information to trigger safer behavior in complex environments.

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

It has been suggested that dopamine (DA) represents reward-prediction-error (RPE) defined in reinforcement learning and therefore DA responds to unpredicted but not predicted reward. However, recent studies have found DA response sustained towards predictable reward in tasks involving self-paced behavior, and suggested that this response represents a motivational signal. We have previously shown that RPE can sustain if there is decay/forgetting of learned-values, which can be implemented as decay of synaptic strengths storing learned-values. This account, however, did not explain the suggested link between tonic/sustained DA and motivation. In the present work, we explored the motivational effects of the value-decay in self-paced approach behavior, modeled as a series of ‘Go’ or ‘No-Go’ selections towards a goal. Through simulations, we found that the value-decay can enhance motivation, specifically, facilitate fast goal-reaching, albeit counterintuitively. Mathematical analyses revealed that underlying potential mechanisms are twofold: (1) decay-induced sustained RPE creates a gradient of ‘Go’ values towards a goal, and (2) value-contrasts between ‘Go’ and ‘No-Go’ are generated because while chosen values are continually updated, unchosen values simply decay. Our model provides potential explanations for the key experimental findings that suggest DA's roles in motivation: (i) slowdown of behavior by post-training blockade of DA signaling, (ii) observations that DA blockade severely impairs effortful actions to obtain rewards while largely sparing seeking of easily obtainable rewards, and (iii) relationships between the reward amount, the level of motivation reflected in the speed of behavior, and the average level of DA. These results indicate that reinforcement learning with value-decay, or forgetting, provides a parsimonious mechanistic account for the DA's roles in value-learning and motivation. Our results also suggest that when biological systems for value-learning are active even though learning has apparently converged, the systems might be in a state of dynamic equilibrium, where learning and forgetting are balanced.

Dopamine (DA) has been suggested to have two reward-related roles: (1) representing reward-prediction-error (RPE), and (2) providing motivational drive. Role(1) is based on the physiological results that DA responds to unpredicted but not predicted reward, whereas role(2) is supported by the pharmacological results that blockade of DA signaling causes motivational impairments such as slowdown of self-paced behavior. So far, these two roles are considered to be played by two different temporal patterns of DA signals: role(1) by phasic signals and role(2) by tonic/sustained signals. However, recent studies have found sustained DA signals with features indicative of both roles (1) and (2), complicating this picture. Meanwhile, whereas synaptic/circuit mechanisms for role(1), i.e., how RPE is calculated in the upstream of DA neurons and how RPE-dependent update of learned-values occurs through DA-dependent synaptic plasticity, have now become clarified, mechanisms for role(2) remain unclear. In this work, we modeled self-paced behavior by a series of ‘Go’ or ‘No-Go’ selections in the framework of reinforcement-learning assuming DA's role(1), and demonstrated that incorporation of decay/forgetting of learned-values, which is presumably implemented as decay of synaptic strengths storing learned-values, provides a potential unified mechanistic account for the DA's two roles, together with its various temporal patterns.

Challenges and promises for translating computational tools into clinical practice

Computational modeling and associated methods have greatly advanced our understanding of cognition and neurobiology underlying complex behaviors and psychiatric conditions. Yet, no computational methods have been successfully translated into clinical settings. This review discusses three major methodological and practical challenges (A. precise characterization of latent neurocognitive processes, B. developing optimal assays, C. developing large-scale longitudinal studies and generating predictions from multi-modal data) and potential promises and tools that have been developed in various fields including mathematical psychology, computational neuroscience, computer science, and statistics. We conclude by highlighting a strong need to communicate and collaborate across multiple disciplines.

A Simplified Model of Choice Behavior under Uncertainty

The Iowa Gambling Task (IGT) has been standardized as a clinical assessment tool (Bechara, 2007). Nonetheless, numerous research groups have attempted to modify IGT models to optimize parameters for predicting the choice behavior of normal controls and patients. A decade ago, most researchers considered the expected utility (EU) model (Busemeyer and Stout, 2002) to be the optimal model for predicting choice behavior under uncertainty. However, in recent years, studies have demonstrated that models with the prospect utility (PU) function are more effective than the EU models in the IGT (Ahn et al., 2008). Nevertheless, after some preliminary tests based on our behavioral dataset and modeling, it was determined that the Ahn et al. (2008) PU model is not optimal due to some incompatible results. This study aims to modify the Ahn et al. (2008) PU model to a simplified model and used the IGT performance of 145 subjects as the benchmark data for comparison. In our simplified PU model, the best goodness-of-fit was found mostly as the value of α approached zero. More specifically, we retested the key parameters α, λ, and A in the PU model. Notably, the influence of the parameters α, λ, and A has a hierarchical power structure in terms of manipulating the goodness-of-fit in the PU model. Additionally, we found that the parameters λ and A may be ineffective when the parameter α is close to zero in the PU model. The present simplified model demonstrated that decision makers mostly adopted the strategy of gain-stay loss-shift rather than foreseeing the long-term outcome. However, there are other behavioral variables that are not well revealed under these dynamic-uncertainty situations. Therefore, the optimal behavioral models may not have been found yet. In short, the best model for predicting choice behavior under dynamic-uncertainty situations should be further evaluated.