Reinforcement Learning in Patients With Mood and Anxiety Disorders vs Control Individuals: A Systematic Review and Meta-analysis

Immunity for counterproductive attentional capture by reward signals among individuals with depressive symptoms

OBJECTIVES

This study aimed to investigate the characteristics of attentional capture by reward signals in individuals with depression during classical conditioning.

METHODS

A variant of the additional singleton paradigm was adopted with a high- or low-reward signal as the prominent distracting stimulus. In Experiment 1, 46 participants with depressive symptoms and 46 healthy controls were asked to conduct a keypress response to the visual target. In Experiment 2, 58 participants with depressive symptoms and 58 healthy controls were asked to conduct a fixation response to the visual target.

RESULTS

In the keypress response task, the presence of high-reward signals slowed down the responses of participants in the control group, whereas the response times of individuals with depression were not significantly affected. In the fixation response task, when the high-reward signal was presented, individuals with depression were more likely to choose the target location as the first saccade destination, compared with healthy controls. In addition, individuals with depression exhibited fewer oculomotor capture by high-reward signals than healthy controls, a trait which was closely linked to the enhanced saccadic inhibition.

CONCLUSION

The results of our study indicated that individuals with depression exhibited an abnormality in attentional capture by reward-related conditioned stimuli during classical conditioning.

Old strategies, new environments: Reinforcement Learning on social media

The rise of social media has profoundly altered the social world - introducing new behaviours which can satisfy our social needs. However, it is yet unknown whether human social strategies, which are well-adapted to the offline world we developed in, operate as effectively within this new social environment. Here, we describe how the computational framework of Reinforcement Learning can help us to precisely frame this problem and diagnose where behaviour-environment mismatches emerge. The Reinforcement Learning framework describes a process by which an agent can learn to maximise their long-term reward. Reinforcement Learning, which has proven successful in characterising human social behaviour, consists of three stages: updating expected reward, valuating expected reward by integrating subjective costs such as effort, and selecting an action. Specific social media affordances, such as the quantifiability of social feedback, might interact with the Reinforcement Learning process at each of these stages. In some cases, affordances can exploit Reinforcement Learning biases which are beneficial offline, by violating the environmental conditions under which such biases are optimal - such as when algorithmic personalisation of content interacts with confirmation bias. Characterising the impact of specific aspects of social media through this lens can improve our understanding of how digital environments shape human behaviour. Ultimately, this formal framework could help address pressing open questions about social media use, including its changing role across human development, and its impact on outcomes such as mental health.

Increasing the Construct Validity of Computational Phenotypes of Mental Illness Through Active Inference and Brain Imaging

After more than 30 years since its inception, the utility of brain imaging for understanding and diagnosing mental illnesses is in doubt, receiving well-grounded criticisms from clinical practitioners. Symptom-based correlational approaches have struggled to provide psychiatry with reliable brain-imaging metrics. However, the emergence of computational psychiatry has paved a new path not only for understanding the psychopathology of mental illness but also to provide practical tools for clinical practice in terms of computational metrics, specifically computational phenotypes. However, these phenotypes still lack sufficient test-retest reliability. In this review, we describe recent works revealing that mind and brain-related computational phenotypes show structural (not random) variation over time, longitudinal changes. Furthermore, we show that these findings suggest that understanding the causes of these changes will improve the construct validity of the phenotypes with an ensuing increase in test-retest reliability. We propose that the active inference framework offers a general-purpose approach for causally understanding these longitudinal changes by incorporating brain imaging as observations within partially observable Markov decision processes.

Prefrontal cortex stimulation normalizes deficient adaptive learning from outcome contingencies in low mood

Depression and anxiety are associated with deficits in adjusting learning behaviour to changing outcome contingencies. This is likely to drive and maintain symptoms, for instance, by perpetuating negative biases or a sense of uncontrollability. Normalising such deficits in adaptive learning might therefore be a novel treatment target for affective disorders. The aim of this experimental medicine study was to test whether prefrontal cortex transcranial direct current stimulation (tDCS) could normalise these aberrant learning processes in depressed mood. To test proof-of-concept, we combined tDCS with a decision-making paradigm that manipulates the volatility of reward and punishment associations. 85 participants with low mood received tDCS during (or before) the task. In two sessions participants received real or sham tDCS in counter-balanced order. Compared to healthy controls (n = 40), individuals with low mood showed significantly impaired adjustment of learning rates to the volatility of loss outcomes. Prefrontal tDCS applied during task performance normalised this deficit, by increasing the adjustment of loss learning rates. As predicted, prefrontal tDCS before task performance (control) had no effect. Thus, the effect was cognitive-state dependent. Our study shows, for the first time, that a candidate depression treatment, prefrontal tDCS, when paired with a task, can reverse deficits in adaptive learning from outcome contingencies in low mood. Thus, combining neurostimulation with a concurrent cognitive manipulation is a potential novel strategy to enhance the effect of tDCS in depression treatment.

Does the reliability of computational models truly improve with hierarchical modeling? Some recommendations and considerations for the assessment of model parameter reliability : Reliability of computational model parameters

Computational modeling of behavior is increasingly being adopted as a standard methodology in psychology, cognitive neuroscience, and computational psychiatry. This approach involves estimating parameters in a computational (or cognitive) model that represents the computational processes of the underlying behavior. In this approach, the reliability of the parameter estimates is an important issue. The use of hierarchical (Bayesian) approaches, which place a prior on each model parameter of the individual participants, is thought to improve the reliability of the parameters. However, the characteristics of reliability in parameter estimates, especially when individual-level priors are assumed, as in hierarchical models, have not yet been fully discussed. Furthermore, the suitability of different reliability measures for assessing parameter reliability is not thoroughly understood. In this study, we conduct a systematic examination of these issues through theoretical analysis and numerical simulations, focusing specifically on reinforcement learning models. We note that the heterogeneity in the estimation precision of individual parameters, particularly with priors, can skew reliability measures toward individuals with higher precision. We further note that there are two factors that reduce reliability, namely estimation error and intersession variation in the true parameters, and we discuss how to evaluate these factors separately. Based on the considerations of this study, we present several recommendations and cautions for assessing the reliability of the model parameters.

Predicting adolescent psychopathology from early life factors: A machine learning tutorial

Objective

The successful implementation and interpretation of machine learning (ML) models in epidemiological studies can be challenging without an extensive programming background. We provide a didactic example of machine learning for risk prediction in this study by determining whether early life factors could be useful for predicting adolescent psychopathology.

Methods

In total, 9643 adolescents ages 9-10 from the Adolescent Brain and Cognitive Development (ABCD) Study were included in ML analysis to predict high Child Behavior Checklist (CBCL) scores (i.e., t-scores ≥ 60). ML models were constructed using a series of predictor combinations (prenatal, family history, sociodemographic) across 5 different algorithms. We assessed ML performance through sensitivity, specificity, F1-score, and area under the curve (AUC) metrics.

Results

A total of 1267 adolescents (13.1 %) were found to have high CBCL scores. The best performing algorithms were elastic net and gradient boosted trees. The best performing elastic net models included prenatal and family history factors (Sensitivity 0.654, Specificity 0.713; AUC 0.742, F1-score 0.401) and prenatal, family, history, and sociodemographic factors (Sensitivity 0.668, Specificity 0.704; AUC 0.745, F1-score 0.402). Across all 5 ML algorithms, family history factors (e.g., either parent had nervous breakdowns, trouble holding jobs/fights/police encounters, and counseling for mental issues) and sociodemographic covariates (e.g., maternal age, child's sex, caregiver income and caregiver education) tended to be better predictors of adolescent psychopathology. The most important prenatal predictors were unplanned pregnancy, birth complications, and pregnancy complications.

Conclusion

Our results suggest that inclusion of prenatal, family history, and sociodemographic factors in ML models can generate moderately accurate predictions of adolescent psychopathology. Issues associated with model overfitting, hyperparameter tuning, and system seed setting should be considered throughout model training, testing, and validation. Future early risk predictions models may improve with the inclusion of additional relevant covariates.

Simulated synapse loss induces depression-like behaviors in deep reinforcement learning

Deep Reinforcement Learning is a branch of artificial intelligence that uses artificial neural networks to model reward-based learning as it occurs in biological agents. Here we modify a Deep Reinforcement Learning approach by imposing a suppressive effect on the connections between neurons in the artificial network-simulating the effect of dendritic spine loss as observed in major depressive disorder (MDD). Surprisingly, this simulated spine loss is sufficient to induce a variety of MDD-like behaviors in the artificially intelligent agent, including anhedonia, increased temporal discounting, avoidance, and an altered exploration/exploitation balance. Furthermore, simulating alternative and longstanding reward-processing-centric conceptions of MDD (dysfunction of the dopamine system, altered reward discounting, context-dependent learning rates, increased exploration) does not produce the same range of MDD-like behaviors. These results support a conceptual model of MDD as a reduction of brain connectivity (and thus information-processing capacity) rather than an imbalance in monoamines-though the computational model suggests a possible explanation for the dysfunction of dopamine systems in MDD. Reversing the spine-loss effect in our computational MDD model can lead to rescue of rewarding behavior under some conditions. This supports the search for treatments that increase plasticity and synaptogenesis, and the model suggests some implications for their effective administration.

Reinforcement-Learning-Informed Queries Guide Behavioral Change

Algorithmically defined aspects of reinforcement learning correlate with psychopathology symptoms and change with symptom improvement following cognitive-behavioral therapy (CBT). Separate work in nonclinical samples has shown that varying the structure and statistics of task environments can change learning. Here, we combine these literatures, drawing on CBT-based guided restructuring of thought processes and computationally defined mechanistic targets identified by reinforcement-learning models in depression, to test whether and how verbal queries affect learning processes. Using a parallel-arm design, we tested 1,299 online participants completing a probabilistic reward-learning task while receiving repeated queries about the task environment (11 learning-query arms and one active control arm). Querying participants about reinforcement-learning-related task components altered computational-model-defined learning parameters in directions specific to the target of the query. These effects on learning parameters were consistent across depression-symptom severity, suggesting new learning-based strategies and therapeutic targets for evoking symptom change in mood psychopathology.

Revisiting the role of computational neuroimaging in the era of integrative neuroscience

Computational models have become integral to human neuroimaging research, providing both mechanistic insights and predictive tools for human cognition and behavior. However, concerns persist regarding the ecological validity of lab-based neuroimaging studies and whether their spatiotemporal resolution is not sufficient for capturing neural dynamics. This review aims to re-examine the utility of computational neuroimaging, particularly in light of the growing prominence of alternative neuroscientific methods and the growing emphasis on more naturalistic behaviors and paradigms. Specifically, we will explore how computational modeling can both enhance the analysis of high-dimensional imaging datasets and, conversely, how neuroimaging, in conjunction with other data modalities, can inform computational models through the lens of neurobiological plausibility. Collectively, this evidence suggests that neuroimaging remains critical for human neuroscience research, and when enhanced by computational models, imaging can serve an important role in bridging levels of analysis and understanding. We conclude by proposing key directions for future research, emphasizing the development of standardized paradigms and the integrative use of computational modeling across neuroimaging techniques.

Computational Mechanisms of Learning and Forgetting Differentiate Affective and Substance Use Disorders

Depression and anxiety are common, highly co-morbid conditions associated with a range of learning and decision-making deficits. While the computational mechanisms underlying these deficits have received growing attention, the transdiagnostic vs. diagnosis-specific nature of these mechanisms remains insufficiently characterized. Individuals with affective disorders (iADs; i.e., depression with or without co-morbid anxiety; N=168 and 74, respectively) completed a widely-used decision-making task. To establish diagnostic specificity, we also incorporated data from a sample of individuals with substance use disorders (iSUDs; N=147) and healthy comparisons (HCs; N=54). Computational modeling afforded separate measures of learning and forgetting rates, among other parameters. Compared to HCs, forgetting rates (reflecting recency bias) were elevated in both iADs and iSUDs (p = 0.007, η 2 = 0.022). In contrast, iADs showed faster learning rates for negative outcomes than iSUDs (p = 0.027, η 2 = 0.017), but they did not differ from HCs. Other model parameters associated with learning and information-seeking also showed suggestive relationships with early adversity and impulsivity. Our findings demonstrate distinct differences in learning and forgetting rates between iSUDs, iADs, and HCs, suggesting that different cognitive processes are affected in these conditions. These differences in decision-making processes and their correlations with symptom dimensions suggest that one could specifically develop interventions that target changing forgetting rates and/or learning from negative outcomes. These results pave the way for replication studies to confirm these relationships and establish their clinical implications.

The neuroscience of mental illness: Building toward the future

Mental illnesses arise from dysfunction in the brain. Although numerous extraneural factors influence these illnesses, ultimately, it is the science of the brain that will lead to novel therapies. Meanwhile, our understanding of this complex organ is incomplete, leading to the oft-repeated trope that neuroscience has yet to make significant contributions to the care of individuals with mental illnesses. This review seeks to counter this narrative, using specific examples of how neuroscientific advances have contributed to progress in mental health care in the past and how current achievements set the stage for further progress in the future.

Lack of optimistic bias during social evaluation learning reflects reduced positive self-beliefs in depression and social anxiety, but via distinct mechanisms

Processing social feedback optimistically may maintain positive self-beliefs and stable social relationships. Conversely, a lack of this optimistic bias in depression and social anxiety may perpetuate negative self-beliefs and maintain symptoms. Research investigating this mechanism is scarce, however, and the mechanisms by which depressed and socially anxious individuals respond to social evaluation may also differ. Using a range of computational approaches in two large datasets (mega-analysis of previous studies, n = 450; pre-registered replication study, n = 807), we investigated how depression (PHQ-9) and social anxiety (BFNE) symptoms related to social evaluation learning in a computerized task. Optimistic bias (better learning of positive relative to negative evaluations) was found to be negatively associated with depression and social anxiety. Structural equation models suggested this reflected a heightened sensitivity to negative social feedback in social anxiety, whereas in depression it co-existed with a blunted response to positive social feedback. Computational belief-based learning models further suggested that reduced optimism was driven by less positive trait-like self-beliefs in both depression and social anxiety, with some evidence for a general blunting in belief updating in depression. Recognizing such transdiagnostic similarities and differences in social evaluation learning across disorders may inform approaches to personalizing treatment.

Individuals with anxiety and depression use atypical decision strategies in an uncertain world

Previous studies on reinforcement learning have identified three prominent phenomena: (1) individuals with anxiety or depression exhibit a reduced learning rate compared to healthy subjects; (2) learning rates may increase or decrease in environments with rapidly changing (i.e. volatile) or stable feedback conditions, a phenomenon termed learning rate adaptation; and (3) reduced learning rate adaptation is associated with several psychiatric disorders. In other words, multiple learning rate parameters are needed to account for behavioral differences across participant populations and volatility contexts in this flexible learning rate (FLR) model. Here, we propose an alternative explanation, suggesting that behavioral variation across participant populations and volatile contexts arises from the use of mixed decision strategies. To test this hypothesis, we constructed a mixture-of-strategies (MOS) model and used it to analyze the behaviors of 54 healthy controls and 32 patients with anxiety and depression in volatile reversal learning tasks. Compared to the FLR model, the MOS model can reproduce the three classic phenomena by using a single set of strategy preference parameters without introducing any learning rate differences. In addition, the MOS model can successfully account for several novel behavioral patterns that cannot be explained by the FLR model. Preferences for different strategies also predict individual variations in symptom severity. These findings underscore the importance of considering mixed strategy use in human learning and decision-making and suggest atypical strategy preference as a potential mechanism for learning deficits in psychiatric disorders.

Reduction of Aversive Learning Rates in Pavlovian Conditioning by Angiotensin II Antagonist Losartan: A Randomized Controlled Trial

BACKGROUND

Angiotensin receptor blockade has been linked to aspects of aversive learning and memory formation and to the prevention of posttraumatic stress disorder symptom development.

METHODS

We investigated the influence of the angiotensin receptor blocker losartan on aversive Pavlovian conditioning using a probabilistic learning paradigm. In a double-blind, randomized, placebo-controlled design, we tested 45 (18 female) healthy volunteers during a baseline session, after application of losartan or placebo (drug session), and during a follow-up session. During each session, participants engaged in a task in which they had to predict the probability of an electrical stimulation on every trial while the true shock contingencies switched repeatedly between phases of high and low shock threat. Computational reinforcement learning models were used to investigate learning dynamics.

RESULTS

Acute administration of losartan significantly reduced participants' adjustment during both low-to-high and high-to-low threat changes. This was driven by reduced aversive learning rates in the losartan group during the drug session compared with baseline. The 50-mg drug dose did not induce reduction of blood pressure or change in reaction times, ruling out a general reduction in attention and engagement. Decreased adjustment of aversive expectations was maintained at a follow-up session 24 hours later.

CONCLUSIONS

This study shows that losartan acutely reduces Pavlovian learning in aversive environments, thereby highlighting a potential role of the renin-angiotensin system in anxiety development.

Spontaneous instrumental approach-avoidance learning in social contexts in autism

BACKGROUND

Individuals with Autism Spectrum Condition (ASC) are characterized by atypicalities in social interactions, compared to Typically Developing individuals (TD). The social motivation theory posits that these difficulties stem from diminished anticipation, reception, and/or learning from social rewards. Although learning from socioemotional outcomes is core to the theory, studies to date have been sparse and inconsistent. This possibly arises from a combination of theoretical, methodological and sample-related issues. Here, we assessed participants' ability to develop a spontaneous preference for actions that lead to desirable socioemotional outcomes (approaching/avoiding of happy/angry individuals, respectively), in an ecologically valid social scenario. We expected that learning abilities would be impaired in ASC individuals, particularly in response to affiliative social feedback.

METHOD

We ran an online social reinforcement learning task, on two large online cohorts with (n = 274) and without (n = 290) ASC, matched for gender, age and education. Participants had to indicate where they would sit in a waiting room. Each seat was associated with different probabilities of approaching/avoiding emotional individuals. Importantly, the task was implicit, as participants were not instructed to learn, and emotional expressions were never mentioned. We applied both categorical analyses contrasting the ASC and TD groups and dimensional factor analysis on affective questionnaires.

RESULTS

Contrary to our hypothesis, participants showed spontaneous learning from socioemotional outcomes, regardless of their diagnostic group. Yet, when accounting for dimensional variations in autistic traits, as well as depression and anxiety, two main findings emerged among females who failed to develop explicit learning strategies: (1) autism severity in ASC correlated with reduced learning to approach happy individuals; (2) anxiety-depression severity across both ASC and TD participants correlated with reduced learning to approach/avoid happy/angry individuals, respectively.

CONCLUSIONS

Implicit spontaneous learning from socioemotional outcomes is not generally impaired in autism but may be specifically associated with autism severity in females with ASC, when they do not have an explicit strategy for adapting to their social environment. Clinical diagnosis and intervention ought to take into account individual differences in their full complexity, including the presence of co-morbid anxiety and depression, when dealing with social atypicalities in autism.

The selective serotonin reuptake inhibitor sertraline alters learning from aversive reinforcements in patients with depression: evidence from a randomized controlled trial

BACKGROUND

Selective serotonin reuptake inhibitors (SSRIs) are first-line pharmacological treatments for depression and anxiety. However, little is known about how pharmacological action is related to cognitive and affective processes. Here, we examine whether specific reinforcement learning processes mediate the treatment effects of SSRIs.

METHODS

The PANDA trial was a multicentre, double-blind, randomized clinical trial in UK primary care comparing the SSRI sertraline with placebo for depression and anxiety. Participants (N = 655) performed an affective Go/NoGo task three times during the trial and computational models were used to infer reinforcement learning processes.

RESULTS

There was poor task performance: only 54% of the task runs were informative, with more informative task runs in the placebo than in the active group. There was no evidence for the preregistered hypothesis that Pavlovian inhibition was affected by sertraline. Exploratory analyses revealed that in the sertraline group, early increases in Pavlovian inhibition were associated with improvements in depression after 12 weeks. Furthermore, sertraline increased how fast participants learned from losses and faster learning from losses was associated with more severe generalized anxiety symptoms.

CONCLUSIONS

The study findings indicate a relationship between aversive reinforcement learning mechanisms and aspects of depression, anxiety, and SSRI treatment, but these relationships did not align with the initial hypotheses. Poor task performance limits the interpretability and likely generalizability of the findings, and highlights the critical importance of developing acceptable and reliable tasks for use in clinical studies.

FUNDING

This article presents research supported by NIHR Program Grants for Applied Research (RP-PG-0610-10048), the NIHR BRC, and UCL, with additional support from IMPRS COMP2PSYCH (JM, QH) and a Wellcome Trust grant (QH).

Uncertainty of treatment efficacy moderates placebo effects on reinforcement learning

The placebo-reward hypothesis postulates that positive effects of treatment expectations on health (i.e., placebo effects) and reward processing share common neural underpinnings. Moreover, experiments in humans and animals indicate that reward uncertainty increases striatal dopamine, which is presumably involved in placebo responses and reward learning. Therefore, treatment uncertainty analogously to reward uncertainty may affect updating from rewards after placebo treatment. Here, we address whether different degrees of uncertainty regarding the efficacy of a sham treatment affect reward sensitivity. In an online between-subjects experiment with N = 141 participants, we systematically varied the provided efficacy instructions before participants first received a sham treatment that consisted of listening to binaural beats and then performed a probabilistic reinforcement learning task. We fitted a Q-learning model including two different learning rates for positive (gain) and negative (loss) reward prediction errors and an inverse gain parameter to behavioral decision data in the reinforcement learning task. Our results yielded an inverted-U-relationship between provided treatment efficacy probability and learning rates for gain, such that higher levels of treatment uncertainty, rather than of expected net efficacy, affect presumably dopamine-related reward learning. These findings support the placebo-reward hypothesis and suggest harnessing uncertainty in placebo treatment for recovering reward learning capabilities.

Directed exploration is elevated in affective disorders but reduced by an aversive interoceptive state induction

Elevated anxiety and uncertainty avoidance are known to exacerbate maladaptive choice in individuals with affective disorders. However, the differential roles of state vs. trait anxiety remain unclear, and underlying computational mechanisms have not been thoroughly characterized. In the present study, we investigated how a somatic (interoceptive) state anxiety induction influences learning and decision-making under uncertainty in individuals with clinically significant levels of trait anxiety. A sample of 58 healthy comparisons (HCs) and 61 individuals with affective disorders (iADs; i.e., depression and/or anxiety) completed a previously validated explore-exploit decision task, with and without an added breathing resistance manipulation designed to induce state anxiety. Computational modeling revealed a pattern in which iADs showed greater information-seeking (i.e., directed exploration; Cohen's d=.39, p=.039) in resting conditions, but that this was reduced by the anxiety induction. The affective disorders group also showed slower learning rates across conditions (Cohen's d=.52, p=.003), suggesting more persistent uncertainty. These findings highlight a complex interplay between trait anxiety and state anxiety. Specifically, while elevated trait anxiety is associated with persistent uncertainty, acute somatic anxiety can paradoxically curtail exploratory behaviors, potentially reinforcing maladaptive decision-making patterns in affective disorders.

The computational structure of consummatory anhedonia

Anhedonia is a reduction in enjoyment, motivation, or interest. It is common across mental health disorders and a harbinger of poor treatment outcomes. The enjoyment aspect, termed 'consummatory anhedonia', in particular poses fundamental questions about how the brain constructs rewards: what processes determine how intensely a reward is experienced? Here, we outline limitations of existing computational conceptualisations of consummatory anhedonia. We then suggest a richer reinforcement learning (RL) account of consummatory anhedonia with a reconceptualisation of subjective hedonic experience in terms of goal progress. This accounts qualitatively for the impact of stress, dysfunctional cognitions, and maladaptive beliefs on hedonic experience. The model also offers new views on the treatments for anhedonia.

Heterogeneity in strategy use during arbitration between experiential and observational learning

To navigate our complex social world, it is crucial to deploy multiple learning strategies, such as learning from directly experiencing action outcomes or from observing other people's behavior. Despite the prevalence of experiential and observational learning in humans and other social animals, it remains unclear how people favor one strategy over the other depending on the environment, and how individuals vary in their strategy use. Here, we describe an arbitration mechanism in which the prediction errors associated with each learning strategy influence their weight over behavior. We designed an online behavioral task to test our computational model, and found that while a substantial proportion of participants relied on the proposed arbitration mechanism, there was some meaningful heterogeneity in how people solved this task. Four other groups were identified: those who used a fixed mixture between the two strategies, those who relied on a single strategy and non-learners with irrelevant strategies. Furthermore, groups were found to differ on key behavioral signatures, and on transdiagnostic symptom dimensions, in particular autism traits and anxiety. Together, these results demonstrate how large heterogeneous datasets and computational methods can be leveraged to better characterize individual differences.

Reward Sensitivity and Noise Contribute to Negative Affective Bias: A Learning Signal Detection Theory Approach in Decision-Making

In patients with mood disorders, negative affective biases - systematically prioritising and interpreting information negatively - are common. A translational cognitive task testing this bias has shown that depressed patients have a reduced preference for a high reward under ambiguous decision-making conditions. The precise mechanisms underscoring this bias are, however, not yet understood. We therefore developed a set of measures to probe the underlying source of the behavioural bias by testing its relationship to a participant's reward sensitivity, value sensitivity and reward learning rate. One-hundred-forty-eight participants completed three online behavioural tasks: the original ambiguous-cue decision-making task probing negative affective bias, a probabilistic reward learning task probing reward sensitivity and reward learning rate, and a gambling task probing value sensitivity. We modelled the learning task through a dynamic signal detection theory model and the gambling task through an expectation-maximisation prospect theory model. Reward sensitivity from the probabilistic reward task (β = 0.131, p = 0.024) and setting noise from the probabilistic reward task (β = -0.187, p = 0.028) both predicted the affective bias score in a logistic regression. Increased negative affective bias, at least on this specific task, may therefore be driven in part by a combination of reduced sensitivity to rewards and more variable responses.

A primer on the use of computational modelling to investigate affective states, affective disorders and animal welfare in non-human animals

Objective measures of animal emotion-like and mood-like states are essential for preclinical studies of affective disorders and for assessing the welfare of laboratory and other animals. However, the development and validation of measures of these affective states poses a challenge partly because the relationships between affect and its behavioural, physiological and cognitive signatures are complex. Here, we suggest that the crisp characterisations offered by computational modelling of the underlying, but unobservable, processes that mediate these signatures should provide better insights. Although this computational psychiatry approach has been widely used in human research in both health and disease, translational computational psychiatry studies remain few and far between. We explain how building computational models with data from animal studies could play a pivotal role in furthering our understanding of the aetiology of affective disorders, associated affective states and the likely underlying cognitive processes involved. We end by outlining the basic steps involved in a simple computational analysis.

Decomposition of Reinforcement Learning Deficits in Disordered Gambling via Drift Diffusion Modeling and Functional Magnetic Resonance Imaging

Gambling disorder is associated with deficits in reward-based learning, but the underlying computational mechanisms are still poorly understood. Here, we examined this issue using a stationary reinforcement learning task in combination with computational modeling and functional resonance imaging (fMRI) in individuals that regular participate in gambling (n = 23, seven fulfilled one to three DSM 5 criteria for gambling disorder, sixteen fulfilled four or more) and matched controls (n = 23). As predicted, the gambling group exhibited substantially reduced accuracy, whereas overall response times (RTs) were not reliably different between groups. We then used comprehensive modeling using reinforcement learning drift diffusion models (RLDDMs) in combination with hierarchical Bayesian parameter estimation to shed light on the computational underpinnings of this performance deficit. In both groups, an RLDDM in which both non-decision time and decision threshold (boundary separation) changed over the course of the experiment accounted for the data best. The model showed good parameter and model recovery, and posterior predictive checks revealed that, in both groups, the model accurately reproduced the evolution of accuracies and RTs over time. Modeling revealed that, compared to controls, the learning impairment in the gambling group was linked to a more rapid reduction in decision thresholds over time, and a reduced impact of value-differences on the drift rate. The gambling group also showed shorter non-decision times. FMRI analyses replicated effects of prediction error coding in the ventral striatum and value coding in the ventro-medial prefrontal cortex, but there was no credible evidence for group differences in these effects. Taken together, our findings show that reinforcement learning impairments in disordered gambling are linked to both maladaptive decision threshold adjustments and a reduced consideration of option values in the choice process.

Impact of AI-Powered Solutions in Rehabilitation Process: Recent Improvements and Future Trends

Rehabilitation is an important and necessary part of local and global healthcare services along with treatment and palliative care, prevention of disease, and promotion of good health. The rehabilitation process helps older and young adults even children to become as independent as possible in activities of daily life and enables participation in useful living activities, recreation, work, and education. The technology of Artificial Intelligence (AI) has evolved significantly in recent years. Many activities related to rehabilitation have been getting benefits from using AI techniques. The objective of this review study is to explore the advantages of AI for rehabilitation and how AI is impacting the rehabilitation process. This study aims at the most critical aspects of the rehabilitation process that could potentially take advantage of AI techniques including personalized rehabilitation apps, rehabilitation through assistance, rehabilitation for neurological disorders, rehabilitation for developmental disorders, virtual reality rehabilitation, rehabilitation of neurodegenerative diseases and Telerehabilitation of Cardiovascular. We presented a survey on the newest empirical studies available in the literature including the AI-based technology helpful in the Rehabilitation process. The novelty feature included but was not limited to an overview of the technological solutions useful in rehabilitation. Seven different categories were identified. Illustrative examples of practical applications were detailed. Implications of the findings for both research and practice were critically discussed. Most of the AI applications in these rehabilitation types are in their infancy and continue to grow while exploring new opportunities. Therefore, we investigate the role of AI technology in rehabilitation processes. In addition, we do statistical analysis of the selected studies to highlight the significance of this review work. In the end, we also present a discussion on some challenges, and future research directions.

Aberrant reward learning, but not negative reinforcement learning, is related to depressive symptoms: an attentional perspective

BACKGROUND

Aberrant reward functioning is implicated in depression. While attention precedes behavior and guides higher-order cognitive processes, reward learning from an attentional perspective - the effects of prior reward-learning on subsequent attention allocation - has been mainly overlooked.

METHODS

The present study explored the effects of reward-based attentional learning in depression using two separate, yet complimentary, studies. In study 1, participants with high (HD) and low (LD) levels of depression symptoms were trained to divert their gaze toward one type of stimuli over another using a novel gaze-contingent music reward paradigm - music played when fixating the desired stimulus type and stopped when gazing the alternate one. Attention allocation was assessed before, during, and following training. In study 2, using negative reinforcement, the same attention allocation pattern was trained while substituting the appetitive music reward for gazing the desired stimulus type with the removal of an aversive sound (i.e. white noise).

RESULTS

In study 1 both groups showed the intended shift in attention allocation during training (online reward learning), while generalization of learning at post-training was only evident among LD participants. Conversely, in study 2 both groups showed post-training generalization. Results were maintained when introducing anxiety as a covariate, and when using a more powerful sensitivity analysis. Finally, HD participants showed higher learning speed than LD participants during initial online learning, but only when using negative, not positive, reinforcement.

CONCLUSIONS

Deficient generalization of learning characterizes the attentional system of HD individuals, but only when using reward-based positive reinforcement, not negative reinforcement.

Perceptual Observer Modeling Reveals Likely Mechanisms of Face Expression Recognition Deficits in Depression

BACKGROUND

Deficits in face emotion recognition are well documented in depression, but the underlying mechanisms are poorly understood. Psychophysical observer models provide a way to precisely characterize such mechanisms. Using model-based analyses, we tested 2 hypotheses about how depression may reduce sensitivity to detect face emotion: 1) via a change in selectivity for visual information diagnostic of emotion or 2) via a change in signal-to-noise ratio in the system performing emotion detection.

METHODS

Sixty adults, one half meeting criteria for major depressive disorder and the other half healthy control participants, identified sadness and happiness in noisy face stimuli, and their responses were used to estimate templates encoding the visual information used for emotion identification. We analyzed these templates using traditional and model-based analyses; in the latter, the match between templates and stimuli, representing sensory evidence for the information encoded in the template, was compared against behavioral data.

RESULTS

Estimated happiness templates produced sensory evidence that was less strongly correlated with response times in participants with depression than in control participants, suggesting that depression was associated with a reduced signal-to-noise ratio in the detection of happiness. The opposite results were found for the detection of sadness. We found little evidence that depression was accompanied by changes in selectivity (i.e., information used to detect emotion), but depression was associated with a stronger influence of face identity on selectivity.

CONCLUSIONS

Depression is more strongly associated with changes in signal-to-noise ratio during emotion recognition, suggesting that deficits in emotion detection are driven primarily by deprecated signal quality rather than suboptimal sampling of information used to detect emotion.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

Depression is associated with reduced outcome sensitivity in a dual valence, magnitude learning task

BACKGROUND

Learning from rewarded and punished choices is perturbed in depressed patients, suggesting that abnormal reinforcement learning may be a cognitive mechanism of the illness. However, previous studies have disagreed about whether this behavior is produced by alterations in the rate of learning or sensitivity to experienced outcomes. This previous work has generally assessed learning in response to binary outcomes of one valence, rather than to both rewarding and punishing continuous outcomes.

METHODS

A novel drifting reward and punishment magnitude reinforcement-learning task was administered to patients with current (n = 40) and remitted depression (n = 39), and healthy volunteers (n = 40) to capture potential differences in learning behavior. Standard questionnaires were administered to measure self-reported depressive symptom severity, trait and state anxiety and level of anhedonic symptoms.

RESULTS

Our findings demonstrate that patients with current depression adjust their learning behaviors to a lesser degree in response to trial-by-trial variations in reward and loss magnitudes than the other groups. Computational modeling revealed that this behavioral signature of current depressive state is better accounted for by reduced reward and punishment sensitivity (all p < 0.031), rather than a change in learning rate (p = 0.708). However, between-group differences were not related to self-reported symptom severity or comorbid anxiety disorders in the current depression group.

CONCLUSION

These findings suggest that current depression is associated with reduced outcome sensitivity rather than altered learning rate. Previous findings reported in this domain mainly from binary learning tasks seem to generalize to learning from continuous outcomes.

Interval Timing as a Computational Pathway From Early Life Adversity to Affective Disorders

Adverse early life experiences can have remarkably enduring negative consequences on mental health, with numerous, varied psychiatric conditions sharing this developmental origin. Yet, the mechanisms linking adverse experiences to these conditions remain poorly understood. Here, we draw on a principled model of interval timing to propose that statistically optimal adaptation of temporal representations to an unpredictable early life environment can produce key characteristics of anhedonia, a transdiagnostic symptom associated with affective disorders like depression and anxiety. The core observation is that early temporal unpredictability produces broader, more imprecise temporal expectations. As a result, reward anticipation is diminished, and associative learning is slowed. When agents with such representations are later introduced to more stable environments, they demonstrate a negativity bias, responding more to the omission of reward than its receipt. Increased encoding of negative events has been proposed to contribute to disorders with anhedonia as a symptom. We then examined how unpredictability interacts with another form of adversity, low reward availability. We found that unpredictability's effect was most strongly felt in richer environments, potentially leading to categorically different phenotypic expressions. In sum, our formalization suggests a single mechanism can help to link early life adversity to a range of behaviors associated with anhedonia, and offers novel insights into the interactive impacts of multiple adversities.

A core component of psychological therapy causes adaptive changes in computational learning mechanisms

BACKGROUND

Cognitive distancing is an emotion regulation strategy commonly used in psychological treatment of various mental health disorders, but its therapeutic mechanisms are unknown.

METHODS

935 participants completed an online reinforcement learning task involving choices between pairs of symbols with differing reward contingencies. Half (49.1%) of the sample was randomised to a cognitive self-distancing intervention and were trained to regulate or 'take a step back' from their emotional response to feedback throughout. Established computational (Q-learning) models were then fit to individuals' choices to derive reinforcement learning parameters capturing clarity of choice values (inverse temperature) and their sensitivity to positive and negative feedback (learning rates).

RESULTS

Cognitive distancing improved task performance, including when participants were later tested on novel combinations of symbols without feedback. Group differences in computational model-derived parameters revealed that cognitive distancing resulted in clearer representations of option values (estimated 0.17 higher inverse temperatures). Simultaneously, distancing caused increased sensitivity to negative feedback (estimated 19% higher loss learning rates). Exploratory analyses suggested this resulted from an evolving shift in strategy by distanced participants: initially, choices were more determined by expected value differences between symbols, but as the task progressed, they became more sensitive to negative feedback, with evidence for a difference strongest by the end of training.

CONCLUSIONS

Adaptive effects on the computations that underlie learning from reward and loss may explain the therapeutic benefits of cognitive distancing. Over time and with practice, cognitive distancing may improve symptoms of mental health disorders by promoting more effective engagement with negative information.

Leveraging Decision Science to Characterize Depression

This brief review examines the potential to use decision science to objectively characterize depression. We provide a brief overview of the existing literature examining different domains of decision-making in depression. Because this overview highlights the specific role of reinforcement learning as an important decision process affected in the disorder, we then introduce reinforcement learning modeling and explain how this approach has identified specific reinforcement learning deficits in depression. We conclude with ideas for future research at the intersection of decision science and depression, emphasizing the potential for decision science to help uncover underlying mechanisms and targets for the treatment of depression.

A Semiparametric Inverse Reinforcement Learning Approach to Characterize Decision Making for Mental Disorders

Major depressive disorder (MDD) is one of the leading causes of disability-adjusted life years. Emerging evidence indicates the presence of reward processing abnormalities in MDD. An important scientific question is whether the abnormalities are due to reduced sensitivity to received rewards or reduced learning ability. Motivated by the probabilistic reward task (PRT) experiment in the EMBARC study, we propose a semiparametric inverse reinforcement learning (RL) approach to characterize the reward-based decision-making of MDD patients. The model assumes that a subject's decision-making process is updated based on a reward prediction error weighted by the subject-specific learning rate. To account for the fact that one favors a decision leading to a potentially high reward, but this decision process is not necessarily linear, we model reward sensitivity with a non-decreasing and nonlinear function. For inference, we estimate the latter via approximation by I-splines and then maximize the joint conditional log-likelihood. We show that the resulting estimators are consistent and asymptotically normal. Through extensive simulation studies, we demonstrate that under different reward-generating distributions, the semiparametric inverse RL outperforms the parametric inverse RL. We apply the proposed method to EMBARC and find that MDD and control groups have similar learning rates but different reward sensitivity functions. There is strong statistical evidence that reward sensitivity functions have nonlinear forms. Using additional brain imaging data in the same study, we find that both reward sensitivity and learning rate are associated with brain activities in the negative affect circuitry under an emotional conflict task.

Approach-avoidance reinforcement learning as a translational and computational model of anxiety-related avoidance

Although avoidance is a prevalent feature of anxiety-related psychopathology, differences in the measurement of avoidance between humans and non-human animals hinder our progress in its theoretical understanding and treatment. To address this, we developed a novel translational measure of anxiety-related avoidance in the form of an approach-avoidance reinforcement learning task, by adapting a paradigm from the non-human animal literature to study the same cognitive processes in human participants. We used computational modelling to probe the putative cognitive mechanisms underlying approach-avoidance behaviour in this task and investigated how they relate to subjective task-induced anxiety. In a large online study (n = 372), participants who experienced greater task-induced anxiety avoided choices associated with punishment, even when this resulted in lower overall reward. Computational modelling revealed that this effect was explained by greater individual sensitivities to punishment relative to rewards. We replicated these findings in an independent sample (n = 627) and we also found fair-to-excellent reliability of measures of task performance in a sub-sample retested 1 week later (n = 57). Our findings demonstrate the potential of approach-avoidance reinforcement learning tasks as translational and computational models of anxiety-related avoidance. Future studies should assess the predictive validity of this approach in clinical samples and experimental manipulations of anxiety.

Anxiety associated with perceived uncontrollable stress enhances expectations of environmental volatility and impairs reward learning

Unavoidable stress can lead to perceived lack of control and learned helplessness, a risk factor for depression. Avoiding punishment and gaining rewards involve updating the values of actions based on experience. Such updating is however useful only if action values are sufficiently stable, something that a lack of control may impair. We examined whether self-reported stress uncontrollability during the first wave of the COVID-19 pandemic predicted impaired reward-learning. In a preregistered study during the first-wave of the COVID-19 pandemic, we used self-reported measures of depression, anxiety, uncontrollable stress, and COVID-19 risk from 427 online participants to predict performance in a three-armed-bandit probabilistic reward learning task. As hypothesised, uncontrollable stress predicted impaired learning, and a greater proportion of probabilistic errors following negative feedback for correct choices, an effect mediated by state anxiety. A parameter from the best-fitting hidden Markov model that estimates expected beliefs that the identity of the optimal choice will shift across images, mediated effects of state anxiety on probabilistic errors and learning deficits. Our findings show that following uncontrollable stress, anxiety promotes an overly volatile representation of the reward-structure of uncertain environments, impairing reward attainment, which is a potential path to anhedonia in depression.

Reduced effects of social feedback on learning in Turner syndrome

Turner syndrome is a genetic condition caused by a complete or partial loss of one of the X chromosomes. Previous studies indicate that Turner syndrome is associated with challenges in social skills, but the underlying mechanisms remain largely unexplored. A possible mechanism is a reduced social influence on learning. The current study examined the impact of social and non-social feedback on learning in women with Turner syndrome (n = 35) and a sex- and age-matched control group (n = 37). Participants were instructed to earn points by repeatedly choosing between two stimuli with unequal probabilities of resulting in a reward. Mastering the task therefore required participants to learn through feedback which of the two stimuli was more likely to be rewarded. Data were analyzed using computational modeling and analyses of choice behavior. Social feedback led to a more explorative choice behavior in the control group, resulting in reduced learning compared to non-social feedback. No effects of social feedback on learning were found in Turner syndrome. The current study thus indicates that women with Turner syndrome may be less sensitive to social influences on reinforcement learning, than the general population.

From Classical Methods to Generative Models: Tackling the Unreliability of Neuroscientific Measures in Mental Health Research

Advances in computational statistics and corresponding shifts in funding initiatives over the past few decades have led to a proliferation of neuroscientific measures being developed in the context of mental health research. Although such measures have undoubtedly deepened our understanding of neural mechanisms underlying cognitive, affective, and behavioral processes associated with various mental health conditions, the clinical utility of such measures remains underwhelming. Recent commentaries point toward the poor reliability of neuroscientific measures to partially explain this lack of clinical translation. Here, we provide a concise theoretical overview of how unreliability impedes clinical translation of neuroscientific measures; discuss how various modeling principles, including those from hierarchical and structural equation modeling frameworks, can help to improve reliability; and demonstrate how to combine principles of hierarchical and structural modeling within the generative modeling framework to achieve more reliable, generalizable measures of brain-behavior relationships for use in mental health research.

The shadowing effect of initial expectation on learning asymmetry

Evidence for positivity and optimism bias abounds in high-level belief updates. However, no consensus has been reached regarding whether learning asymmetries exist in more elementary forms of updates such as reinforcement learning (RL). In RL, the learning asymmetry concerns the sensitivity difference in incorporating positive and negative prediction errors (PE) into value estimation, namely the asymmetry of learning rates associated with positive and negative PEs. Although RL has been established as a canonical framework in characterizing interactions between agent and environment, the direction of learning asymmetry remains controversial. Here, we propose that part of the controversy stems from the fact that people may have different value expectations before entering the learning environment. Such a default value expectation influences how PEs are calculated and consequently biases subjects' choices. We test this hypothesis in two learning experiments with stable or varying reinforcement probabilities, across monetary gains, losses, and gain-loss mixed environments. Our results consistently support the model incorporating both asymmetric learning rates and the initial value expectation, highlighting the role of initial expectation in value updating and choice preference. Further simulation and model parameter recovery analyses confirm the unique contribution of initial value expectation in accessing learning rate asymmetry.

Contextual influence of reinforcement learning performance of depression: evidence for a negativity bias?

BACKGROUNDS

Value-based decision-making impairment in depression is a complex phenomenon: while some studies did find evidence of blunted reward learning and reward-related signals in the brain, others indicate no effect. Here we test whether such reward sensitivity deficits are dependent on the overall value of the decision problem.

METHODS

We used a two-armed bandit task with two different contexts: one 'rich', one 'poor' where both options were associated with an overall positive, negative expected value, respectively. We tested patients (N = 30) undergoing a major depressive episode and age, gender and socio-economically matched controls (N = 26). Learning performance followed by a transfer phase, without feedback, were analyzed to distangle between a decision or a value-update process mechanism. Finally, we used computational model simulation and fitting to link behavioral patterns to learning biases.

RESULTS

Control subjects showed similar learning performance in the 'rich' and the 'poor' contexts, while patients displayed reduced learning in the 'poor' context. Analysis of the transfer phase showed that the context-dependent impairment in patients generalized, suggesting that the effect of depression has to be traced to the outcome encoding. Computational model-based results showed that patients displayed a higher learning rate for negative compared to positive outcomes (the opposite was true in controls).

CONCLUSIONS

Our results illustrate that reinforcement learning performances in depression depend on the value of the context. We show that depressive patients have a specific trouble in contexts with an overall negative state value, which in our task is consistent with a negativity bias at the learning rates level.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Sensitivity and specificity in affective and social learning in adolescence

Adolescence is a period of heightened affective and social sensitivity. In this review we address how this increased sensitivity influences associative learning. Based on recent evidence from human and rodent studies, as well as advances in computational biology, we suggest that, compared to other age groups, adolescents show features of heightened Pavlovian learning but tend to perform worse than adults at instrumental learning. Because Pavlovian learning does not involve decision-making, whereas instrumental learning does, we propose that these developmental differences might be due to heightened sensitivity to rewards and threats in adolescence, coupled with a lower specificity of responding. We discuss the implications of these findings for adolescent mental health and education.

Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty

People radically differ in how they cope with uncertainty. Clinical researchers describe a dispositional characteristic known as "intolerance of uncertainty", a tendency to find uncertainty aversive, reported to be elevated across psychiatric and neurodevelopmental conditions. Concurrently, recent research in computational psychiatry has leveraged theoretical work to characterise individual differences in uncertainty processing. Under this framework, differences in how people estimate different forms of uncertainty can contribute to mental health difficulties. In this review, we briefly outline the concept of intolerance of uncertainty within its clinical context, and we argue that the mechanisms underlying this construct may be further elucidated through modelling how individuals make inferences about uncertainty. We will review the evidence linking psychopathology to different computationally specified forms of uncertainty and consider how these findings might suggest distinct mechanistic routes towards intolerance of uncertainty. We also discuss the implications of this computational approach for behavioural and pharmacological interventions, as well as the importance of different cognitive domains and subjective experiences in studying uncertainty processing.

Spatiotemporal Precision of Neuroimaging in Psychiatry

Aberrant patterns of cognition, perception, and behavior seen in psychiatric disorders are thought to be driven by a complex interplay of neural processes that evolve at a rapid temporal scale. Understanding these dynamic processes in vivo in humans has been hampered by a trade-off between spatial and temporal resolutions inherent to current neuroimaging technology. A recent trend in psychiatric research has been the use of high temporal resolution imaging, particularly magnetoencephalography, often in conjunction with sophisticated machine learning decoding techniques. Developments here promise novel insights into the spatiotemporal dynamics of cognitive phenomena, including domains relevant to psychiatric illnesses such as reward and avoidance learning, memory, and planning. This review considers recent advances afforded by exploiting this increased spatiotemporal precision, with specific reference to applications that seek to drive a mechanistic understanding of psychopathology and the realization of preclinical translation.

Computational Mechanisms of Addiction and Anxiety: A Developmental Perspective

A central goal of computational psychiatry is to identify systematic relationships between transdiagnostic dimensions of psychiatric symptomatology and the latent learning and decision-making computations that inform individuals' thoughts, feelings, and choices. Most psychiatric disorders emerge prior to adulthood, yet little work has extended these computational approaches to study the development of psychopathology. Here, we lay out a roadmap for future studies implementing this approach by developing empirically and theoretically informed hypotheses about how developmental changes in model-based control of action and Pavlovian learning processes may modulate vulnerability to anxiety and addiction. We highlight how insights from studies leveraging computational approaches to characterize the normative developmental trajectories of clinically relevant learning and decision-making processes may suggest promising avenues for future developmental computational psychiatry research.

Belief updating in psychosis, depression and anxiety disorders: A systematic review across computational modelling approaches

Alterations in belief updating are proposed to underpin symptoms of psychiatric illness, including psychosis, depression, and anxiety. Key parameters underlying belief updating can be captured using computational modelling techniques, aiding the identification of unique and shared deficits, and improving diagnosis and treatment. We systematically reviewed research that applied computational modelling to probabilistic tasks measuring belief updating in stable and volatile (changing) environments, across clinical and subclinical psychosis (n = 17), anxiety (n = 9), depression (n = 9) and transdiagnostic samples (n = 9). Depression disorders related to abnormal belief updating in response to the valence of rewards, evidenced in both stable and volatile environments. Whereas psychosis and anxiety disorders were associated with difficulties adapting to changing contingencies specifically, indicating an inflexibility and/or insensitivity to environmental volatility. Higher-order learning models revealed additional difficulties in the estimation of overall environmental volatility across psychosis disorders, showing increased updating to irrelevant information. These findings stress the importance of investigating belief updating in transdiagnostic samples, using homogeneous experimental and computational modelling approaches.

Individual differences in computational psychiatry: a review of current challenges

Bringing precision to the understanding and treatment of mental disorders requires instruments for studying clinically relevant individual differences. One promising approach is the development of computational assays: integrating computational models with cognitive tasks to infer latent patient-specific disease processes in brain computations. While recent years have seen many methodological advancements in computational modelling and many cross-sectional patient studies, much less attention has been paid to basic psychometric properties (reliability and construct validity) of the computational measures provided by the assays. In this review, we assess the extent of this issue by examining emerging empirical evidence. We find that many computational measures suffer from poor psychometric properties, which poses a risk of invalidating previous findings and undermining ongoing research efforts using computational assays to study individual (and even group) differences. We provide recommendations for how to address these problems and, crucially, embed them within a broader perspective on key developments that are needed for translating computational assays to clinical practice.

Stimulating human prefrontal cortex increases reward learning

Work in computational psychiatry suggests that mood disorders may stem from aberrant reinforcement learning processes. Specifically, it has been proposed that depressed individuals believe that negative events are more informative than positive events, resulting in higher learning rates from negative outcomes (Pulcu and Browning, 2019). In this proof-of-concept study, we investigated whether transcranial direct current stimulation (tDCS) applied to dorsolateral prefrontal cortex, as commonly used in depression treatment trials, might change learning rates for affective outcomes. Healthy adults completed an established reinforcement learning task (Pulcu and Browning, 2017) in which the information content of reward and loss outcomes was manipulated by varying the volatility of stimulus-outcome associations. Learning rates on the tasks were quantified using computational models. Stimulation over dorsolateral prefrontal cortex (DLPFC) but not motor cortex (M1) increased learning rates specifically for reward outcomes. The effects of prefrontal tDCS were cognitive state-dependent: tDCS applied during task performance increased learning rates for wins; tDCS applied before task performance decreased both win and loss learning rates. A replication study confirmed the key finding that tDCS to DLPFC during task performance increased learning rates specifically for rewards. Taken together, these findings demonstrate the potential of tDCS for modulating computational parameters of reinforcement learning that are relevant to mood disorders.

Anhedonia and sensitivity to punishment in schizophrenia, depression and opiate use disorder

BACKGROUND

From a behavioural perspective anhedonia is defined as diminished interest in the engagement of pleasurable activities. Despite its presence across a range of psychiatric disorders, the cognitive processes that give rise to anhedonia remain unclear.

METHODS

Here we examine whether anhedonia is associated with learning from positive and negative outcomes in patients diagnosed with major depression, schizophrenia and opiate use disorder alongside a healthy control group. Responses in the Wisconsin Card Sorting Test - a task associated with healthy prefrontal cortex function - were fitted to the Attentional Learning Model (ALM) which separates learning from positive and negative feedback.

RESULTS

Learning from punishment, but not from reward, was negatively associated with anhedonia beyond other socio-demographic, cognitive and clinical variables. This impairment in punishment sensitivity was also associated with faster responses following negative feedback, independently of the degree of surprise.

LIMITATIONS

Future studies should test the longitudinal association between punishment sensitivity and anhedonia also in other clinical populations controlling for the effect of specific medications.

CONCLUSIONS

Together the results reveal that anhedonic subjects, because of their negative expectations, are less sensitive to negative feedbacks; this might lead them to persist in actions leading to negative outcomes.

Formalising social representation to explain psychiatric symptoms

Recent work in social cognition has moved beyond a focus on how people process social rewards to examine how healthy people represent other agents and how this is altered in psychiatric disorders. However, formal modelling of social representation has not kept pace with these changes, impeding our understanding of how core aspects of social cognition function, and fail, in psychopathology. Here, we suggest that belief-based computational models provide a basis for an integrated sociocognitive approach to psychiatry, with the potential to address important but unexamined pathologies of social representation, such as maladaptive schemas and illusory social agents.