Imprecise action selection in substance use disorder: Evidence for active learning impairments when solving the explore-exploit dilemma

Individuals with Methamphetamine Use Disorder Show Reduced Directed Exploration and Learning Rates Independent of an Aversive Interoceptive State Induction

Methamphetamine Use Disorder (MUD) is associated with substantially reduced quality of life. Yet, decisions to use persist, due in part to avoidance of anticipated withdrawal states. However, the specific cognitive mechanisms underlying this decision process, and possible modulatory effects of aversive states, remain unclear. Here, 56 individuals with MUD and 58 healthy comparisons (HCs) performed a decision task, both with and without an aversive interoceptive state induction. Computational modeling measured the tendency to test beliefs about uncertain outcomes (directed exploration) and the ability to update beliefs in response to outcomes (learning rates). Compared to HCs, those with MUD exhibited less directed exploration and slower learning rates, but these differences were not affected by aversive state induction. These results suggest novel, state-independent computational mechanisms whereby individuals with MUD may have difficulties in testing beliefs about the tolerability of abstinence and in adjusting behavior in response to consequences of continued use.

Dysfunctional feedback processing in male methamphetamine abusers: Evidence from neurophysiological and computational approaches

Methamphetamine use disorder (MUD) as a major public health risk is associated with dysfunctional neural feedback processing. Although dysfunctional feedback processing in people who are substance dependent has been explored in several behavioral, computational, and electrocortical studies, this mechanism in MUDs requires to be well understood. Furthermore, the current understanding of latent components of their behavior such as learning speed and exploration-exploitation dilemma is still limited. In addition, the association between the latent cognitive components and the related neural mechanisms also needs to be explored. Therefore, in this study, the underlying neurocognitive mechanisms of feedback processing of such impairment, and age/gender-matched healthy controls are evaluated within a probabilistic learning task with rewards and punishments. Mathematical modeling results based on the Q-learning paradigm suggested that MUDs show less sensitivity in distinguishing optimal options. Additionally, it may be worth noting that MUDs exhibited a slight decrease in their ability to learn from negative feedback compared to healthy controls. Also through the lens of underlying neural mechanisms, MUDs showed lower theta power at the medial-frontal areas while responding to negative feedback. However, other EEG measures of reinforcement learning including feedback-related negativity, parietal-P300, and activity flow from the medial frontal to lateral prefrontal regions, remained intact in MUDs. On the other hand, the elimination of the linkage between value sensitivity and medial-frontal theta activity in MUDs was observed. The observed dysfunction could be due to the adverse effects of methamphetamine on the cortico-striatal dopamine circuit, which is reflected in the anterior cingulate cortex activity as the most likely region responsible for efficient behavior adjustment. These findings could help us to pave the way toward tailored therapeutic approaches.

Reviewing explore/exploit decision-making as a transdiagnostic target for psychosis, depression, and anxiety

In many everyday decisions, individuals choose between trialling something novel or something they know well. Deciding when to try a new option or stick with an option that is already known to you, known as the "explore/exploit" dilemma, is an important feature of cognition that characterises a range of decision-making contexts encountered by humans. Recent evidence has suggested preferences in explore/exploit biases are associated with psychopathology, although this has typically been examined within individual disorders. The current review examined whether explore/exploit decision-making represents a promising transdiagnostic target for psychosis, depression, and anxiety. A systematic search of academic databases was conducted, yielding a total of 29 studies. Studies examining psychosis were mostly consistent in showing that individuals with psychosis explored more compared with individuals without psychosis. The literature on anxiety and depression was more heterogenous; some studies found that anxiety and depression were associated with more exploration, whereas other studies demonstrated reduced exploration in anxiety and depression. However, examining a subset of studies that employed case-control methods, there was some evidence that both anxiety and depression also were associated with increased exploration. Due to the heterogeneity across the literature, we suggest that there is insufficient evidence to conclude whether explore/exploit decision-making is a transdiagnostic target for psychosis, depression, and anxiety. However, alongside our advisory groups of lived experience advisors, we suggest that this context of decision-making is a promising candidate that merits further investigation using well-powered, longitudinal designs. Such work also should examine whether biases in explore/exploit choices are amenable to intervention.

People with a tobacco use disorder exhibit misaligned Bayesian belief updating by falsely attributing non-drug cues as worse predictors of positive outcomes compared to drug cues

Adaptive behaviours depend on dynamically updating internal representations of the world based on the ever-changing environmental contingencies. People with a substance use disorder (pSUD) show maladaptive behaviours with high persistence in drug-taking, despite severe negative consequences. We recently proposed a salience misattribution model for addiction (SMMA; Kalhan et al., 2021), arguing that pSUD have aberrations in their updating processes where drug cues are misattributed as strong predictors of positive outcomes, but weaker predictors of negative outcomes. We also argued that conversely, non-drug cues are misattributed as weak predictors of positive outcomes, but stronger predictors of negative outcomes. We tested these hypotheses using a multi-cue reversal learning task, with reversals in whether drug or non-drug cues are relevant in predicting the outcome (monetary win or loss). We show that people with a tobacco use disorder (pTUD), do form misaligned internal representations. We found that pTUD updated less towards learning the drug cue's relevance in predicting a loss. Further, when neither drug nor non-drug cue predicted a win, pTUD updated more towards the drug cue being relevant predictors of that win. Our Bayesian belief updating model revealed that pTUD had a low estimated likelihood of non-drug cues being predictors of wins, compared to drug cues, which drove the misaligned updating. Overall, several hypotheses of the SMMA were supported, but not all. Our results implicate that strengthening the non-drug cue association with positive outcomes may help restore the misaligned internal representation in pTUD, and offers a quantifiable, computational account of these updating processes.

The empirical status of predictive coding and active inference

Research on predictive processing models has focused largely on two specific algorithmic theories: Predictive Coding for perception and Active Inference for decision-making. While these interconnected theories possess broad explanatory potential, they have only recently begun to receive direct empirical evaluation. Here, we review recent studies of Predictive Coding and Active Inference with a focus on evaluating the degree to which they are empirically supported. For Predictive Coding, we find that existing empirical evidence offers modest support. However, some positive results can also be explained by alternative feedforward (e.g., feature detection-based) models. For Active Inference, most empirical studies have focused on fitting these models to behavior as a means of identifying and explaining individual or group differences. While Active Inference models tend to explain behavioral data reasonably well, there has not been a focus on testing empirical validity of active inference theory per se, which would require formal comparison to other models (e.g., non-Bayesian or model-free reinforcement learning models). This review suggests that, while promising, a number of specific research directions are still necessary to evaluate the empirical adequacy and explanatory power of these algorithms.

Pre-exposure Prophylaxis Adherence Among Men Who Have Sex With Men: A Systematic Review and Meta-analysis

OBJECTIVES

The effectiveness and efficiency of pre-exposure prophylaxis (PrEP) in reducing the transmission of human immunodeficiency virus (HIV) among men who have sex with men (MSM) relies on how widely it is adopted and adhered to, particularly among high-risk groups of MSM. The meta-analysis aimed to collect and analyze existing evidence on various factors related to PrEP adherence in MSM, including demographic characteristics, sexual behaviors, substance use, and psychosocial factors.

METHODS

The meta-analysis followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The search included articles published between January 2018 and December 2022, obtained from the PubMed, ScienceDirect, and Scopus databases. The studies that were included in the analysis reported the proportion of MSM who demonstrated adherence to PrEP and underwent quality appraisal using the Newcastle-Ottawa Scale.

RESULTS

Of the 268 studies initially identified, only 12 met the inclusion criteria and were included in the final meta-analysis. The findings indicated that education (odds ratio [OR], 1.64; 95% confidence interval [CI], 1.12 to 2.40), number of sexual partners (OR, 1.16; 95% CI, 1.02 to 1.31), engaging in sexual activities with an human immunodeficiency virus-positive partner (OR, 1.59; 95% CI, 1.16 to 2.26), substance use (OR, 0.83; 95% CI, 0.70 to 0.99), and lower levels of depression (OR, 0.55; 95% CI, 0.37 to 0.82) were associated with higher rates of PrEP adherence among MSM.

CONCLUSIONS

Despite these findings, further research is necessary to investigate PrEP adherence more comprehensively. The findings of this meta-analysis can be utilized to inform interventions aimed at improving PrEP adherence among MSM and provide directions for future research in this area.

Active learning impairments in substance use disorders when resolving the explore-exploit dilemma: A replication and extension of previous computational modeling results

BACKGROUND

Substance use disorders (SUDs) represent a major public health risk. Yet, our understanding of the mechanisms that maintain these disorders remains incomplete. In a recent computational modeling study, we found initial evidence that SUDs are associated with slower learning rates from negative outcomes and less value-sensitive choice (low "action precision"), which could help explain continued substance use despite harmful consequences.

METHODS

Here we aimed to replicate and extend these results in a pre-registered study with a new sample of 168 individuals with SUDs and 99 healthy comparisons (HCs). We performed the same computational modeling and group comparisons as in our prior report (doi: 10.1016/j.drugalcdep.2020.108208) to confirm previously observed effects. After completing all pre-registered replication analyses, we then combined the previous and current datasets (N = 468) to assess whether differences were transdiagnostic or driven by specific disorders.

RESULTS

Replicating prior results, SUDs showed slower learning rates for negative outcomes in both Bayesian and frequentist analyses (partial η2=.02). Previously observed differences in action precision were not confirmed. Learning rates for positive outcomes were also similar between groups. Logistic regressions including all computational parameters as predictors in the combined datasets could differentiate several specific disorders from HCs, but could not differentiate most disorders from each other.

CONCLUSIONS

These results provide robust evidence that individuals with SUDs adjust behavior more slowly in the face of negative outcomes than HCs. They also suggest this effect is common across several different SUDs. Future research should examine its neural basis and whether learning rates could represent a new treatment target or moderator of treatment outcome.

An Introduction to Predictive Processing Models of Perception and Decision-Making

The predictive processing framework includes a broad set of ideas, which might be articulated and developed in a variety of ways, concerning how the brain may leverage predictive models when implementing perception, cognition, decision-making, and motor control. This article provides an up-to-date introduction to the two most influential theories within this framework: predictive coding and active inference. The first half of the paper (Sections 2-5) reviews the evolution of predictive coding, from early ideas about efficient coding in the visual system to a more general model encompassing perception, cognition, and motor control. The theory is characterized in terms of the claims it makes at Marr's computational, algorithmic, and implementation levels of description, and the conceptual and mathematical connections between predictive coding, Bayesian inference, and variational free energy (a quantity jointly evaluating model accuracy and complexity) are explored. The second half of the paper (Sections 6-8) turns to recent theories of active inference. Like predictive coding, active inference models assume that perceptual and learning processes minimize variational free energy as a means of approximating Bayesian inference in a biologically plausible manner. However, these models focus primarily on planning and decision-making processes that predictive coding models were not developed to address. Under active inference, an agent evaluates potential plans (action sequences) based on their expected free energy (a quantity that combines anticipated reward and information gain). The agent is assumed to represent the world as a partially observable Markov decision process with discrete time and discrete states. Current research applications of active inference models are described, including a range of simulation work, as well as studies fitting models to empirical data. The paper concludes by considering future research directions that will be important for further development of both models.

Lifeworlds in pain: a principled method for investigation and intervention

The experience of pain spans biological, psychological and sociocultural realms, both basic and complex, it is by turns necessary and devastating. Despite an extensive knowledge of the constituents of pain, the ability to translate this into effective intervention remains limited. It is suggested that current, multiscale, medical approaches, largely informed by the biopsychosocial (BPS) model, attempt to integrate knowledge but are undermined by an epistemological obligation, one that necessitates a prior isolation of the constituent parts. To overcome this impasse, we propose that an anthropological stance needs to be taken, underpinned by a Bayesian apparatus situated in computational psychiatry. Here, pain is presented within the context of lifeworlds, where attention is shifted away from the constituents of experience (e.g. nociception, reward processing and fear-avoidance), towards the dynamic affiliation that occurs between these processes over time. We argue that one can derive a principled method of investigation and intervention for pain from modelling approaches in computational psychiatry. We suggest that these modelling methods provide the necessary apparatus to navigate multiscale ontology and epistemology of pain. Finally, a unified approach to the experience of pain is presented, where the relational, inter-subjective phenomenology of pain is brought into contact with a principled method of translation; in so doing, revealing the conditions and possibilities of lifeworlds in pain.

Barriers and solutions to the adoption of translational tools for computational psychiatry

Computational psychiatry is a field aimed at developing formal models of information processing in the human brain, and how alterations in this processing can lead to clinical phenomena. There has been significant progress in the development of tasks and how to model them, presenting an opportunity to incorporate computational psychiatry methodologies into large- scale research projects or into clinical practice. In this viewpoint, we explore some of the barriers to incorporation of computational psychiatry tasks and models into wider mainstream research directions. These barriers include the time required for participants to complete tasks, test-retest reliability, limited ecological validity, as well as practical concerns, such as lack of computational expertise and the expense and large sample sizes traditionally required to validate tasks and models. We then discuss solutions, such as the redesigning of tasks with a view toward feasibility, and the integration of tasks into more ecologically valid and standardized game platforms that can be more easily disseminated. Finally, we provide an example of how one task, the conditioned hallucinations task, might be translated into such a game. It is our hope that interest in the creation of more accessible and feasible computational tasks will help computational methods make more positive impacts on research as well as, eventually, clinical practice.

Computational models of behavioral addictions: State of the art and future directions

Non-pharmacological behavioral addictions, such as pathological gambling, videogaming, social networking, or internet use, are becoming major public health concerns. It is not yet clear how behavioral addictions could share many major neurobiological and behavioral characteristics with substance use disorders, despite the absence of direct pharmacological influences. A deeper understanding of the neurocognitive mechanisms of addictive behavior is needed, and computational modeling could be one promising approach to explain intricately entwined cognitive and neural dynamics. This review describes computational models of addiction based on reinforcement learning algorithms, Bayesian inference, and biophysical neural simulations. We discuss whether computational frameworks originally conceived to explain maladaptive behavior in substance use disorders can be effectively extended to non-substance-related behavioral addictions. Moreover, we introduce recent studies on behavioral addictions that exemplify the possibility of such extension and propose future directions.

Elevated decision uncertainty and reduced avoidance drives in depression, anxiety and substance use disorders during approach-avoidance conflict: a replication study

BACKGROUND

Decision-making under approach-avoidance conflict (AAC; e.g., sacrificing quality of life to avoid feared outcomes) may be affected in multiple psychiatric disorders. Recently, we used a computational (active inference) model to characterize information processing differences during AAC in individuals with depression, anxiety and/or substance use disorders. Individuals with psychiatric disorders exhibited increased decision uncertainty (DU) and reduced sensitivity to unpleasant stimuli. This preregistered study aimed to determine the replicability of this processing dysfunction.

METHODS

A new sample of participants completed the AAC task. Individual-level computational parameter estimates, reflecting decision uncertainty and sensitivity to unpleasant stimuli ("emotion conflict"; EC), were obtained and compared between groups. Subsequent analyses combining the prior and current samples allowed assessment of narrower disorder categories.

RESULTS

The sample in the present study included 480 participants: 97 healthy controls, 175 individuals with substance use disorders and 208 individuals with depression and/or anxiety disorders. Individuals with substance use disorders showed higher DU and lower EC values than healthy controls. The EC values were lower in females, but not males, with depression and/or anxiety disorders than in healthy controls. However, the previously observed difference in DU between participants with depression and/or anxiety disorders and healthy controls did not replicate. Analyses of specific disorders in the combined samples indicated that effects were common across different substance use disorders and affective disorders.

LIMITATIONS

There were differences, although with small effect size, in age and baseline intellectual functioning between the previous and current sample, which may have affected replication of DU differences in participants with depression and/or anxiety disorders.

CONCLUSION

The now robust evidence base for these clinical group differences motivates specific questions that should be addressed in future research: can DU and EC become behavioural treatment targets, and can we identify neural substrates of DU and EC that could be used to measure severity of dysfunction or as neuromodulatory treatment targets?

Active learning impairments in substance use disorders when resolving the explore-exploit dilemma: A replication and extension of previous computational modeling results

Background

Substance use disorders (SUDs) represent a major public health risk. Yet, our understanding of the mechanisms that maintain these disorders remains incomplete. In a recent computational modeling study, we found initial evidence that SUDs are associated with slower learning rates from negative outcomes and less value-sensitive choice (low "action precision"), which could help explain continued substance use despite harmful consequences.

Methods

Here we aimed to replicate and extend these results in a pre-registered study with a new sample of 168 individuals with SUDs and 99 healthy comparisons (HCs). We performed the same computational modeling and group comparisons as in our prior report (doi: 10.1016/j.drugalcdep.2020.108208) to confirm previously observed effects. After completing all pre-registered replication analyses, we then combined the previous and current datasets (N = 468) to assess whether differences were transdiagnostic or driven by specific disorders.

Results

Replicating prior results, SUDs showed slower learning rates for negative outcomes in both Bayesian and frequentist analyses (η 2 =.02). Previously observed differences in action precision were not confirmed. Logistic regressions including all computational parameters as predictors in the combined datasets could differentiate several specific disorders from HCs, but could not differentiate most disorders from each other.

Conclusions

These results provide robust evidence that individuals with SUDs have more difficulty adjusting behavior in the face of negative outcomes than HCs. They also suggest this effect is common across several different SUDs. Future research should examine its neural basis and whether learning rates could represent a new treatment target or moderator of treatment outcome.

Computational mechanisms underpinning greater exploratory behaviour in excess weight relative to healthy weight adolescents

Obesity in adolescence is associated with cognitive changes that lead to difficulties in shifting unhealthy habits in favour of alternative healthy behaviours, similar to addictive behaviours. An outstanding question is whether this shift in goal-directed behaviour is driven by over-exploitation or over-exploration of rewarding outcomes. Here, we addressed this question by comparing explore/exploit behaviour on the Iowa Gambling Task in 43 adolescents with excess weight against 38 adolescents with healthy weight. We computationally modelled both exploitation behaviour (e.g., reinforcement sensitivity and inverse decay parameters), and explorative behaviour (e.g., maximum directed exploration value). We found that overall, adolescents with excess weight displayed more behavioural exploration than their healthy-weight counterparts - specifically, demonstrating greater overall switching behaviour. Computational models revealed that this behaviour was driven by a higher maximum directed exploration value in the excess-weight group (U = 520.00, p = .005, BF₁₀ = 5.11). Importantly, however, we found substantial evidence that groups did not differ in reinforcement sensitivity (U = 867.00, p = .641, BF₁₀ = 0.30). Overall, our study demonstrates a preference for exploratory behaviour in adolescents with excess weight, independent of sensitivity to reward. This pattern could potentially underpin an intrinsic desire to explore energy-dense unhealthy foods - an as-yet untapped mechanism that could be targeted in future treatments of obesity in adolescents.

Active inference and the two-step task

Sequential decision problems distill important challenges frequently faced by humans. Through repeated interactions with an uncertain world, unknown statistics need to be learned while balancing exploration and exploitation. Reinforcement learning is a prominent method for modeling such behaviour, with a prevalent application being the two-step task. However, recent studies indicate that the standard reinforcement learning model sometimes describes features of human task behaviour inaccurately and incompletely. We investigated whether active inference, a framework proposing a trade-off to the exploration-exploitation dilemma, could better describe human behaviour. Therefore, we re-analysed four publicly available datasets of the two-step task, performed Bayesian model selection, and compared behavioural model predictions. Two datasets, which revealed more model-based inference and behaviour indicative of directed exploration, were better described by active inference, while the models scored similarly for the remaining datasets. Learning using probability distributions appears to contribute to the improved model fits. Further, approximately half of all participants showed sensitivity to information gain as formulated under active inference, although behavioural exploration effects were not fully captured. These results contribute to the empirical validation of active inference as a model of human behaviour and the study of alternative models for the influential two-step task.

Slower Learning Rates from Negative Outcomes in Substance Use Disorder over a 1-Year Period and Their Potential Predictive Utility

Computational modelling is a promising approach to parse dysfunctional cognitive processes in substance use disorders (SUDs), but it is unclear how much these processes change during the recovery period. We assessed 1-year follow-up data on a sample of treatment-seeking individuals with one or more SUDs (alcohol, cannabis, sedatives, stimulants, hallucinogens, and/or opioids; N = 83) that were previously assessed at baseline within a prior computational modelling study. Relative to healthy controls (HCs; N = 48), these participants were found at baseline to show altered learning rates and less precise action selection while completing an explore-exploit decision-making task. Here we replicated these analyses when these individuals returned and re-performed the task 1 year later to assess the stability of baseline differences. We also examined whether baseline modelling measures could predict symptoms at follow-up. Bayesian and frequentist analyses indicated that: (a) group differences in learning rates were stable over time (posterior probability = 1); and (b) intra-class correlations (ICCs) between model parameters at baseline and follow-up were significant and ranged from small to moderate (.25 ≤ ICCs ≤ .54). Exploratory analyses also suggested that learning rates and/or information-seeking values at baseline were associated with substance use severity at 1-year follow-up in stimulant and opioid users (.36 ≤ rs ≤ .43). These findings suggest that learning dysfunctions are moderately stable during recovery and could correspond to trait-like vulnerability factors. In addition, computational measures at baseline had some predictive value for changes in substance use severity over time and could be clinically informative.

Computational models of exploration and exploitation characterise onset and efficacy of treatment in methamphetamine use disorder

People with Methamphetamine Use Disorder (PwMUD) spend substantial time and resources on substance use, which hinders their ability to explore alternate reinforcers. Gold‐standard behavioural treatments attempt to remedy this by encouraging action towards non‐drug reinforcers, but substance use often persists. We aimed to unravel the mechanistic drivers of this behaviour by applying a computational model of explore/exploit behaviour to decision‐making data (Iowa Gambling Task) from 106 PwMUD and 48 controls. We then examined the longitudinal link between explore/exploit mechanisms and changes in methamphetamine use 6 weeks later. Exploitation parameters included reinforcement sensitivity and inverse decay (i.e., number of past outcomes used to guide choices). Exploration parameters included maximum directed exploration value (i.e., value of trying novel actions). The Timeline Follow Back measured changes in methamphetamine use. Compared to controls, PwMUD showed deficits in exploitative decision‐making, characterised by reduced reinforcement sensitivity, U = 3065, p = 0.009, and less use of previous choice outcomes, U = 3062, p = 0.010. This was accompanied by a behavioural pattern of frequent shifting between choices, which appeared consistent with random exploration. Furthermore, PwMUD with greater reductions of methamphetamine use at 6 weeks had increased directed exploration (β = 0.22, p = 0.045); greater use of past choice outcomes (β = −0.39, p = 0.002) and greater choice consistency (β = −0.39, p = 0.002). Therefore, limited computational exploitation and increased behavioural exploration characterise PwMUD's presentation to treatment, while increased directed exploration, use of past choice outcomes and choice consistency predict greater reductions of methamphetamine use.

A step-by-step tutorial on active inference and its application to empirical data

The active inference framework, and in particular its recent formulation as a partially observable Markov decision process (POMDP), has gained increasing popularity in recent years as a useful approach for modeling neurocognitive processes. This framework is highly general and flexible in its ability to be customized to model any cognitive process, as well as simulate predicted neuronal responses based on its accompanying neural process theory. It also affords both simulation experiments for proof of principle and behavioral modeling for empirical studies. However, there are limited resources that explain how to build and run these models in practice, which limits their widespread use. Most introductions assume a technical background in programming, mathematics, and machine learning. In this paper we offer a step-by-step tutorial on how to build POMDPs, run simulations using standard MATLAB routines, and fit these models to empirical data. We assume a minimal background in programming and mathematics, thoroughly explain all equations, and provide exemplar scripts that can be customized for both theoretical and empirical studies. Our goal is to provide the reader with the requisite background knowledge and practical tools to apply active inference to their own research. We also provide optional technical sections and multiple appendices, which offer the interested reader additional technical details. This tutorial should provide the reader with all the tools necessary to use these models and to follow emerging advances in active inference research.

From Generative Models to Generative Passages: A Computational Approach to (Neuro) Phenomenology

This paper presents a version of neurophenomenology based on generative modelling techniques developed in computational neuroscience and biology. Our approach can be described as computational phenomenology because it applies methods originally developed in computational modelling to provide a formal model of the descriptions of lived experience in the phenomenological tradition of philosophy (e.g., the work of Edmund Husserl, Maurice Merleau-Ponty, etc.). The first section presents a brief review of the overall project to naturalize phenomenology. The second section presents and evaluates philosophical objections to that project and situates our version of computational phenomenology with respect to these projects. The third section reviews the generative modelling framework. The final section presents our approach in detail. We conclude by discussing how our approach differs from previous attempts to use generative modelling to help understand consciousness. In summary, we describe a version of computational phenomenology which uses generative modelling to construct a computational model of the inferential or interpretive processes that best explain this or that kind of lived experience.

The experimental tobacco marketplace: Effects of low-ventilated cigarette exposure

OBJECTIVES

Regulating filter ventilation will change the relative reinforcing value of products resulting in nicotine/tobacco users facing the explore/exploit dilemma (ie, choice between unfamiliar and familiar options). This study examined the effects of price increases in higher-ventilated cigarettes (HVCs) and exposure to lower-ventilated cigarettes (LVCs) on explore/exploit patterns of tobacco-product purchasing in the Experimental Tobacco Marketplace (ETM).

METHODS

HVC smokers (N = 20) completed one assessment session and 3 ETM sessions separated by weeks of at-home LVC exposure. In each ETM session, participants made 7-days of tobacco-product purchases as HVCs price increased across trials.

RESULTS

Prohibitive prices of HVC decreased the likelihood of HVCs purchases and increased the likelihood of LVC purchases. Initial exposure (week 1) to LVC reduced the number of cigarettes purchased when HVC prices were high and increased exploration of alternative tobacco products. Successive exposure to LVC (repeated access in weeks 2,5,6,9,10) decreased likelihood of HVCs and alternative product purchases and increased the likelihood of LVCs purchases.

CONCLUSIONS

Regulating filter ventilation may initially increase exploration of alternative tobacco products but lead to exploitation of LVCs over time. Tobacco control strategies should take advantage of this transition period when smokers seek information on unfamiliar products to implement harm reduction strategies.

Reinforcement learning detuned in addiction: integrative and translational approaches

Suboptimal decision-making strategies have been proposed to contribute to the pathophysiology of addiction. Decision-making, however, arises from a collection of computational components that can independently influence behavior. Disruptions in these different components can lead to decision-making deficits that appear similar behaviorally, but differ at the computational, and likely the neurobiological, level. Here, we discuss recent studies that have used computational approaches to investigate the decision-making processes underlying addiction. Studies in animal models have found that value updating following positive, but not negative, outcomes is predictive of drug use, whereas value updating following negative, but not positive, outcomes is disrupted following drug self-administration. We contextualize these findings with studies on the circuit and biological mechanisms of decision-making to develop a framework for revealing the biobehavioral mechanisms of addiction.

Lower Levels of Directed Exploration and Reflective Thinking Are Associated With Greater Anxiety and Depression

Anxiety and depression are often associated with strong beliefs that entering specific situations will lead to aversive outcomes - even when these situations are objectively safe and avoiding them reduces well-being. A possible mechanism underlying this maladaptive avoidance behavior is a failure to reflect on: (1) appropriate levels of uncertainty about the situation, and (2) how this uncertainty could be reduced by seeking further information (i.e., exploration). To test this hypothesis, we asked a community sample of 416 individuals to complete measures of reflective cognition, exploration, and symptoms of anxiety and depression. Consistent with our hypotheses, we found significant associations between each of these measures in expected directions (i.e., positive relationships between reflective cognition and strategic information-seeking behavior or "directed exploration", and negative relationships between these measures and anxiety/depression symptoms). Further analyses suggested that the relationship between directed exploration and depression/anxiety was due in part to an ambiguity aversion promoting exploration in conditions where information-seeking was not beneficial (as opposed to only being due to under-exploration when more information would aid future choices). In contrast, reflectiveness was associated with greater exploration in appropriate settings and separately accounted for differences in reaction times, decision noise, and choice accuracy in expected directions. These results shed light on the mechanisms underlying information-seeking behavior and how they may contribute to symptoms of emotional disorders. They also highlight the potential clinical relevance of individual differences in reflectiveness and exploration and should motivate future research on their possible contributions to vulnerability and/or maintenance of affective disorders.

Computational analysis of probabilistic reversal learning deficits in male subjects with alcohol use disorder

BACKGROUND

Alcohol use disorder is characterized by perseverative alcohol use despite negative consequences. This hallmark feature of addiction potentially relates to impairments in behavioral flexibility, which can be measured by probabilistic reversal learning (PRL) paradigms. We here aimed to examine the cognitive mechanisms underlying impaired PRL task performance in patients with alcohol use disorder (AUDP) using computational models of reinforcement learning.

METHODS

Twenty-eight early abstinent AUDP and 27 healthy controls (HC) performed an extensive PRL paradigm. We compared conventional behavioral variables of choices (perseveration; correct responses) between groups. Moreover, we fitted Bayesian computational models to the task data to compare differences in latent cognitive variables including reward and punishment learning and choice consistency between groups.

RESULTS

AUDP and HC did not significantly differ with regard to direct perseveration rates after reversals. However, AUDP made overall less correct responses and specifically showed decreased win-stay behavior compared to HC. Interestingly, AUDP showed premature switching after no or little negative feedback but elevated proneness to stay when accumulation of negative feedback would make switching a more optimal option. Computational modeling revealed that AUDP compared to HC showed enhanced learning from punishment, a tendency to learn less from positive feedback and lower choice consistency.

CONCLUSION

Our data do not support the assumption that AUDP are characterized by increased perseveration behavior. Instead our findings provide evidence that enhanced negative reinforcement and decreased non-drug-related reward learning as well as diminished choice consistency underlie dysfunctional choice behavior in AUDP.

An empirical evaluation of active inference in multi-armed bandits

A key feature of sequential decision making under uncertainty is a need to balance between exploiting-choosing the best action according to the current knowledge, and exploring-obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration-exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.

Gut inference: A computational modelling approach

Neurocomputational theories have hypothesized that Bayesian inference underlies interoception, which has become a topic of recent experimental work in heartbeat perception. To extend this approach beyond cardiac interoception, we describe the application of a Bayesian computational model to a recently developed gastrointestinal interoception task completed by 40 healthy individuals undergoing simultaneous electroencephalogram (EEG) and peripheral physiological recording. We first present results that support the validity of this modelling approach. Second, we provide a test of, and confirmatory evidence supporting, the neural process theory associated with a particular Bayesian framework (active inference) that predicts specific relationships between computational parameters and event-related potentials in EEG. We also offer some exploratory evidence suggesting that computational parameters may influence the regulation of peripheral physiological states. We conclude that this computational approach offers promise as a tool for studying individual differences in gastrointestinal interoception.

Long-term stability of computational parameters during approach-avoidance conflict in a transdiagnostic psychiatric patient sample

Maladaptive behavior during approach-avoidance conflict (AAC) is common to multiple psychiatric disorders. Using computational modeling, we previously reported that individuals with depression, anxiety, and substance use disorders (DEP/ANX; SUDs) exhibited differences in decision uncertainty and sensitivity to negative outcomes versus reward (emotional conflict) relative to healthy controls (HCs). However, it remains unknown whether these computational parameters and group differences are stable over time. We analyzed 1-year follow-up data from a subset of the same participants (N = 325) to assess parameter stability and relationships to other clinical and task measures. We assessed group differences in the entire sample as well as a subset matched for age and IQ across HCs (N = 48), SUDs (N = 29), and DEP/ANX (N = 121). We also assessed 2-3 week reliability in a separate sample of 30 HCs. Emotional conflict and decision uncertainty parameters showed moderate 1-year intra-class correlations (.52 and .46, respectively) and moderate to excellent correlations over the shorter period (.84 and .54, respectively). Similar to previous baseline findings, parameters correlated with multiple response time measures (ps < .001) and self-reported anxiety (r = .30, p < .001) and decision difficulty (r = .44, p < .001). Linear mixed effects analyses revealed that patients remained higher in decision uncertainty (SUDs, p = .009) and lower in emotional conflict (SUDs, p = .004, DEP/ANX, p = .02) relative to HCs. This computational modelling approach may therefore offer relatively stable markers of transdiagnostic psychopathology.

Neural Dynamics under Active Inference: Plausibility and Efficiency of Information Processing

Active inference is a normative framework for explaining behaviour under the free energy principle-a theory of self-organisation originating in neuroscience. It specifies neuronal dynamics for state-estimation in terms of a descent on (variational) free energy-a measure of the fit between an internal (generative) model and sensory observations. The free energy gradient is a prediction error-plausibly encoded in the average membrane potentials of neuronal populations. Conversely, the expected probability of a state can be expressed in terms of neuronal firing rates. We show that this is consistent with current models of neuronal dynamics and establish face validity by synthesising plausible electrophysiological responses. We then show that these neuronal dynamics approximate natural gradient descent, a well-known optimisation algorithm from information geometry that follows the steepest descent of the objective in information space. We compare the information length of belief updating in both schemes, a measure of the distance travelled in information space that has a direct interpretation in terms of metabolic cost. We show that neural dynamics under active inference are metabolically efficient and suggest that neural representations in biological agents may evolve by approximating steepest descent in information space towards the point of optimal inference.

Deeply Felt Affect: The Emergence of Valence in Deep Active Inference

The positive-negative axis of emotional valence has long been recognized as fundamental to adaptive behavior, but its origin and underlying function have largely eluded formal theorizing and computational modeling. Using deep active inference, a hierarchical inference scheme that rests on inverting a model of how sensory data are generated, we develop a principled Bayesian model of emotional valence. This formulation asserts that agents infer their valence state based on the expected precision of their action model-an internal estimate of overall model fitness ("subjective fitness"). This index of subjective fitness can be estimated within any environment and exploits the domain generality of second-order beliefs (beliefs about beliefs). We show how maintaining internal valence representations allows the ensuing affective agent to optimize confidence in action selection preemptively. Valence representations can in turn be optimized by leveraging the (Bayes-optimal) updating term for subjective fitness, which we label affective charge (AC). AC tracks changes in fitness estimates and lends a sign to otherwise unsigned divergences between predictions and outcomes. We simulate the resulting affective inference by subjecting an in silico affective agent to a T-maze paradigm requiring context learning, followed by context reversal. This formulation of affective inference offers a principled account of the link between affect, (mental) action, and implicit metacognition. It characterizes how a deep biological system can infer its affective state and reduce uncertainty about such inferences through internal action (i.e., top-down modulation of priors that underwrite confidence). Thus, we demonstrate the potential of active inference to provide a formal and computationally tractable account of affect. Our demonstration of the face validity and potential utility of this formulation represents the first step within a larger research program. Next, this model can be leveraged to test the hypothesized role of valence by fitting the model to behavioral and neuronal responses.

Perceptual insensitivity to the modulation of interoceptive signals in depression, anxiety, and substance use disorders

This study employed a series of heartbeat perception tasks to assess the hypothesis that cardiac interoceptive processing in individuals with depression/anxiety (N = 221), and substance use disorders (N = 136) is less flexible than that of healthy individuals (N = 53) in the context of physiological perturbation. Cardiac interoception was assessed via heartbeat tapping when: (1) guessing was allowed; (2) guessing was not allowed; and (3) experiencing an interoceptive perturbation (inspiratory breath hold) expected to amplify cardiac sensation. Healthy participants showed performance improvements across the three conditions, whereas those with depression/anxiety and/or substance use disorder showed minimal improvement. Machine learning analyses suggested that individual differences in these improvements were negatively related to anxiety sensitivity, but explained relatively little variance in performance. These results reveal a perceptual insensitivity to the modulation of interoceptive signals that was evident across several common psychiatric disorders, suggesting that interoceptive deficits in the realm of psychopathology manifest most prominently during states of homeostatic perturbation.

Greater decision uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: a computational modelling approach

BACKGROUND

Imbalances in approach-avoidance conflict (AAC) decision-making (e.g., sacrificing rewards to avoid negative outcomes) are considered central to multiple psychiatric disorders. We used computational modelling to examine 2 factors that are often not distinguished in descriptive analyses of AAC: decision uncertainty and sensitivity to negative outcomes versus rewards (emotional conflict).

METHODS

A previously validated AAC task was completed by 478 participants, including healthy controls (n = 59), people with substance use disorders (n = 159) and people with depression and/or anxiety disorders who did not have substance use disorders (n = 260). Using an active inference model, we estimated individual-level values for a model parameter that reflected decision uncertainty and another that reflected emotional conflict. We also repeated analyses in a subsample (59 healthy controls, 161 people with depression and/or anxiety disorders, 56 people with substance use disorders) that was propensity-matched for age and general intelligence.

RESULTS

The model showed high accuracy (72%). As further validation, parameters correlated with reaction times and self-reported task motivations in expected directions. The emotional conflict parameter further correlated with self-reported anxiety during the task (r = 0.32, p < 0.001), and the decision uncertainty parameter correlated with self-reported difficulty making decisions (r = 0.45, p < 0.001). Compared to healthy controls, people with depression and/or anxiety disorders and people with substance use disorders showed higher decision uncertainty in the propensity-matched sample (t = 2.16, p = 0.03, and t = 2.88, p = 0.005, respectively), with analogous results in the full sample; people with substance use disorders also showed lower emotional conflict in the full sample (t = 3.17, p = 0.002).

LIMITATIONS

This study was limited by heterogeneity of the clinical sample and an inability to examine learning.

CONCLUSION

These results suggest that reduced confidence in how to act, rather than increased emotional conflict, may explain maladaptive approach-avoidance behaviours in people with psychiatric disorders.

Greater decision uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: a computational modelling approach

BACKGROUND

Imbalances in approach-avoidance conflict (AAC) decision-making (e.g., sacrificing rewards to avoid negative outcomes) are considered central to multiple psychiatric disorders. We used computational modelling to examine 2 factors that are often not distinguished in descriptive analyses of AAC: decision uncertainty and sensitivity to negative outcomes versus rewards (emotional conflict).

METHODS

A previously validated AAC task was completed by 478 participants, including healthy controls (n = 59), people with substance use disorders (n = 159) and people with depression and/or anxiety disorders who did not have substance use disorders (n = 260). Using an active inference model, we estimated individual-level values for a model parameter that reflected decision uncertainty and another that reflected emotional conflict. We also repeated analyses in a subsample (59 healthy controls, 161 people with depression and/or anxiety disorders, 56 people with substance use disorders) that was propensity-matched for age and general intelligence.

RESULTS

The model showed high accuracy (72%). As further validation, parameters correlated with reaction times and self-reported task motivations in expected directions. The emotional conflict parameter further correlated with self-reported anxiety during the task (r = 0.32, p < 0.001), and the decision uncertainty parameter correlated with self-reported difficulty making decisions (r = 0.45, p < 0.001). Compared to healthy controls, people with depression and/or anxiety disorders and people with substance use disorders showed higher decision uncertainty in the propensity-matched sample (t = 2.16, p = 0.03, and t = 2.88, p = 0.005, respectively), with analogous results in the full sample; people with substance use disorders also showed lower emotional conflict in the full sample (t = 3.17, p = 0.002).

LIMITATIONS

This study was limited by heterogeneity of the clinical sample and an inability to examine learning.

CONCLUSION

These results suggest that reduced confidence in how to act, rather than increased emotional conflict, may explain maladaptive approach-avoidance behaviours in people with psychiatric disorders.

Recent advances in the application of predictive coding and active inference models within clinical neuroscience

Research in clinical neuroscience is founded on the idea that a better understanding of brain (dys)function will improve our ability to diagnose and treat neurological and psychiatric disorders. In recent years, neuroscience has converged on the notion that the brain is a 'prediction machine,' in that it actively predicts the sensory input that it will receive if one or another course of action is chosen. These predictions are used to select actions that will (most often, and in the long run) maintain the body within the narrow range of physiological states consistent with survival. This insight has given rise to an area of clinical computational neuroscience research that focuses on characterizing neural circuit architectures that can accomplish these predictive functions, and on how the associated processes may break down or become aberrant within clinical conditions. Here, we provide a brief review of examples of recent work on the application of predictive processing models of brain function to study clinical (psychiatric) disorders, with the aim of highlighting current directions and their potential clinical utility. We offer examples of recent conceptual models, formal mathematical models, and applications of such models in empirical research in clinical populations, with a focus on making this material accessible to clinicians without expertise in computational neuroscience. In doing so, we aim to highlight the potential insights and opportunities that understanding the brain as a prediction machine may offer to clinical research and practice.