Reinforcement Learning in Patients With Mood and Anxiety Disorders vs Control Individuals: A Systematic Review and Meta-analysis

Uncertainty of treatment efficacy moderates placebo effects on reinforcement learning

The placebo-reward hypothesis postulates that positive effects of treatment expectations on health (i.e., placebo effects) and reward processing share common neural underpinnings. Moreover, experiments in humans and animals indicate that reward uncertainty increases striatal dopamine, which is presumably involved in placebo responses and reward learning. Therefore, treatment uncertainty analogously to reward uncertainty may affect updating from rewards after placebo treatment. Here, we address whether different degrees of uncertainty regarding the efficacy of a sham treatment affect reward sensitivity. In an online between-subjects experiment with N = 141 participants, we systematically varied the provided efficacy instructions before participants first received a sham treatment that consisted of listening to binaural beats and then performed a probabilistic reinforcement learning task. We fitted a Q-learning model including two different learning rates for positive (gain) and negative (loss) reward prediction errors and an inverse gain parameter to behavioral decision data in the reinforcement learning task. Our results yielded an inverted-U-relationship between provided treatment efficacy probability and learning rates for gain, such that higher levels of treatment uncertainty, rather than of expected net efficacy, affect presumably dopamine-related reward learning. These findings support the placebo-reward hypothesis and suggest harnessing uncertainty in placebo treatment for recovering reward learning capabilities.

Directed exploration is elevated in affective disorders but reduced by an aversive interoceptive state induction

Elevated anxiety and uncertainty avoidance are known to exacerbate maladaptive choice in individuals with affective disorders. However, the differential roles of state vs. trait anxiety remain unclear, and underlying computational mechanisms have not been thoroughly characterized. In the present study, we investigated how a somatic (interoceptive) state anxiety induction influences learning and decision-making under uncertainty in individuals with clinically significant levels of trait anxiety. A sample of 58 healthy comparisons (HCs) and 61 individuals with affective disorders (iADs; i.e., depression and/or anxiety) completed a previously validated explore-exploit decision task, with and without an added breathing resistance manipulation designed to induce state anxiety. Computational modeling revealed a pattern in which iADs showed greater information-seeking (i.e., directed exploration; Cohen's d=.39, p=.039) in resting conditions, but that this was reduced by the anxiety induction. The affective disorders group also showed slower learning rates across conditions (Cohen's d=.52, p=.003), suggesting more persistent uncertainty. These findings highlight a complex interplay between trait anxiety and state anxiety. Specifically, while elevated trait anxiety is associated with persistent uncertainty, acute somatic anxiety can paradoxically curtail exploratory behaviors, potentially reinforcing maladaptive decision-making patterns in affective disorders.

The computational structure of consummatory anhedonia

Anhedonia is a reduction in enjoyment, motivation, or interest. It is common across mental health disorders and a harbinger of poor treatment outcomes. The enjoyment aspect, termed 'consummatory anhedonia', in particular poses fundamental questions about how the brain constructs rewards: what processes determine how intensely a reward is experienced? Here, we outline limitations of existing computational conceptualisations of consummatory anhedonia. We then suggest a richer reinforcement learning (RL) account of consummatory anhedonia with a reconceptualisation of subjective hedonic experience in terms of goal progress. This accounts qualitatively for the impact of stress, dysfunctional cognitions, and maladaptive beliefs on hedonic experience. The model also offers new views on the treatments for anhedonia.

Heterogeneity in strategy use during arbitration between experiential and observational learning

To navigate our complex social world, it is crucial to deploy multiple learning strategies, such as learning from directly experiencing action outcomes or from observing other people's behavior. Despite the prevalence of experiential and observational learning in humans and other social animals, it remains unclear how people favor one strategy over the other depending on the environment, and how individuals vary in their strategy use. Here, we describe an arbitration mechanism in which the prediction errors associated with each learning strategy influence their weight over behavior. We designed an online behavioral task to test our computational model, and found that while a substantial proportion of participants relied on the proposed arbitration mechanism, there was some meaningful heterogeneity in how people solved this task. Four other groups were identified: those who used a fixed mixture between the two strategies, those who relied on a single strategy and non-learners with irrelevant strategies. Furthermore, groups were found to differ on key behavioral signatures, and on transdiagnostic symptom dimensions, in particular autism traits and anxiety. Together, these results demonstrate how large heterogeneous datasets and computational methods can be leveraged to better characterize individual differences.

Reward Sensitivity and Noise Contribute to Negative Affective Bias: A Learning Signal Detection Theory Approach in Decision-Making

In patients with mood disorders, negative affective biases - systematically prioritising and interpreting information negatively - are common. A translational cognitive task testing this bias has shown that depressed patients have a reduced preference for a high reward under ambiguous decision-making conditions. The precise mechanisms underscoring this bias are, however, not yet understood. We therefore developed a set of measures to probe the underlying source of the behavioural bias by testing its relationship to a participant's reward sensitivity, value sensitivity and reward learning rate. One-hundred-forty-eight participants completed three online behavioural tasks: the original ambiguous-cue decision-making task probing negative affective bias, a probabilistic reward learning task probing reward sensitivity and reward learning rate, and a gambling task probing value sensitivity. We modelled the learning task through a dynamic signal detection theory model and the gambling task through an expectation-maximisation prospect theory model. Reward sensitivity from the probabilistic reward task (β = 0.131, p = 0.024) and setting noise from the probabilistic reward task (β = -0.187, p = 0.028) both predicted the affective bias score in a logistic regression. Increased negative affective bias, at least on this specific task, may therefore be driven in part by a combination of reduced sensitivity to rewards and more variable responses.

A primer on the use of computational modelling to investigate affective states, affective disorders and animal welfare in non-human animals

Objective measures of animal emotion-like and mood-like states are essential for preclinical studies of affective disorders and for assessing the welfare of laboratory and other animals. However, the development and validation of measures of these affective states poses a challenge partly because the relationships between affect and its behavioural, physiological and cognitive signatures are complex. Here, we suggest that the crisp characterisations offered by computational modelling of the underlying, but unobservable, processes that mediate these signatures should provide better insights. Although this computational psychiatry approach has been widely used in human research in both health and disease, translational computational psychiatry studies remain few and far between. We explain how building computational models with data from animal studies could play a pivotal role in furthering our understanding of the aetiology of affective disorders, associated affective states and the likely underlying cognitive processes involved. We end by outlining the basic steps involved in a simple computational analysis.

Decomposition of Reinforcement Learning Deficits in Disordered Gambling via Drift Diffusion Modeling and Functional Magnetic Resonance Imaging

Gambling disorder is associated with deficits in reward-based learning, but the underlying computational mechanisms are still poorly understood. Here, we examined this issue using a stationary reinforcement learning task in combination with computational modeling and functional resonance imaging (fMRI) in individuals that regular participate in gambling (n = 23, seven fulfilled one to three DSM 5 criteria for gambling disorder, sixteen fulfilled four or more) and matched controls (n = 23). As predicted, the gambling group exhibited substantially reduced accuracy, whereas overall response times (RTs) were not reliably different between groups. We then used comprehensive modeling using reinforcement learning drift diffusion models (RLDDMs) in combination with hierarchical Bayesian parameter estimation to shed light on the computational underpinnings of this performance deficit. In both groups, an RLDDM in which both non-decision time and decision threshold (boundary separation) changed over the course of the experiment accounted for the data best. The model showed good parameter and model recovery, and posterior predictive checks revealed that, in both groups, the model accurately reproduced the evolution of accuracies and RTs over time. Modeling revealed that, compared to controls, the learning impairment in the gambling group was linked to a more rapid reduction in decision thresholds over time, and a reduced impact of value-differences on the drift rate. The gambling group also showed shorter non-decision times. FMRI analyses replicated effects of prediction error coding in the ventral striatum and value coding in the ventro-medial prefrontal cortex, but there was no credible evidence for group differences in these effects. Taken together, our findings show that reinforcement learning impairments in disordered gambling are linked to both maladaptive decision threshold adjustments and a reduced consideration of option values in the choice process.

Impact of AI-Powered Solutions in Rehabilitation Process: Recent Improvements and Future Trends

Rehabilitation is an important and necessary part of local and global healthcare services along with treatment and palliative care, prevention of disease, and promotion of good health. The rehabilitation process helps older and young adults even children to become as independent as possible in activities of daily life and enables participation in useful living activities, recreation, work, and education. The technology of Artificial Intelligence (AI) has evolved significantly in recent years. Many activities related to rehabilitation have been getting benefits from using AI techniques. The objective of this review study is to explore the advantages of AI for rehabilitation and how AI is impacting the rehabilitation process. This study aims at the most critical aspects of the rehabilitation process that could potentially take advantage of AI techniques including personalized rehabilitation apps, rehabilitation through assistance, rehabilitation for neurological disorders, rehabilitation for developmental disorders, virtual reality rehabilitation, rehabilitation of neurodegenerative diseases and Telerehabilitation of Cardiovascular. We presented a survey on the newest empirical studies available in the literature including the AI-based technology helpful in the Rehabilitation process. The novelty feature included but was not limited to an overview of the technological solutions useful in rehabilitation. Seven different categories were identified. Illustrative examples of practical applications were detailed. Implications of the findings for both research and practice were critically discussed. Most of the AI applications in these rehabilitation types are in their infancy and continue to grow while exploring new opportunities. Therefore, we investigate the role of AI technology in rehabilitation processes. In addition, we do statistical analysis of the selected studies to highlight the significance of this review work. In the end, we also present a discussion on some challenges, and future research directions.

Aberrant reward learning, but not negative reinforcement learning, is related to depressive symptoms: an attentional perspective

BACKGROUND

Aberrant reward functioning is implicated in depression. While attention precedes behavior and guides higher-order cognitive processes, reward learning from an attentional perspective - the effects of prior reward-learning on subsequent attention allocation - has been mainly overlooked.

METHODS

The present study explored the effects of reward-based attentional learning in depression using two separate, yet complimentary, studies. In study 1, participants with high (HD) and low (LD) levels of depression symptoms were trained to divert their gaze toward one type of stimuli over another using a novel gaze-contingent music reward paradigm - music played when fixating the desired stimulus type and stopped when gazing the alternate one. Attention allocation was assessed before, during, and following training. In study 2, using negative reinforcement, the same attention allocation pattern was trained while substituting the appetitive music reward for gazing the desired stimulus type with the removal of an aversive sound (i.e. white noise).

RESULTS

In study 1 both groups showed the intended shift in attention allocation during training (online reward learning), while generalization of learning at post-training was only evident among LD participants. Conversely, in study 2 both groups showed post-training generalization. Results were maintained when introducing anxiety as a covariate, and when using a more powerful sensitivity analysis. Finally, HD participants showed higher learning speed than LD participants during initial online learning, but only when using negative, not positive, reinforcement.

CONCLUSIONS

Deficient generalization of learning characterizes the attentional system of HD individuals, but only when using reward-based positive reinforcement, not negative reinforcement.

Perceptual Observer Modeling Reveals Likely Mechanisms of Face Expression Recognition Deficits in Depression

BACKGROUND

Deficits in face emotion recognition are well documented in depression, but the underlying mechanisms are poorly understood. Psychophysical observer models provide a way to precisely characterize such mechanisms. Using model-based analyses, we tested 2 hypotheses about how depression may reduce sensitivity to detect face emotion: 1) via a change in selectivity for visual information diagnostic of emotion or 2) via a change in signal-to-noise ratio in the system performing emotion detection.

METHODS

Sixty adults, one half meeting criteria for major depressive disorder and the other half healthy control participants, identified sadness and happiness in noisy face stimuli, and their responses were used to estimate templates encoding the visual information used for emotion identification. We analyzed these templates using traditional and model-based analyses; in the latter, the match between templates and stimuli, representing sensory evidence for the information encoded in the template, was compared against behavioral data.

RESULTS

Estimated happiness templates produced sensory evidence that was less strongly correlated with response times in participants with depression than in control participants, suggesting that depression was associated with a reduced signal-to-noise ratio in the detection of happiness. The opposite results were found for the detection of sadness. We found little evidence that depression was accompanied by changes in selectivity (i.e., information used to detect emotion), but depression was associated with a stronger influence of face identity on selectivity.

CONCLUSIONS

Depression is more strongly associated with changes in signal-to-noise ratio during emotion recognition, suggesting that deficits in emotion detection are driven primarily by deprecated signal quality rather than suboptimal sampling of information used to detect emotion.

Reduction of Aversive Learning Rates in Pavlovian Conditioning by Angiotensin II Antagonist Losartan: A Randomized Controlled Trial

BACKGROUND

Angiotensin receptor blockade has been linked to aspects of aversive learning and memory formation and to the prevention of posttraumatic stress disorder symptom development.

METHODS

We investigated the influence of the angiotensin receptor blocker losartan on aversive Pavlovian conditioning using a probabilistic learning paradigm. In a double-blind, randomized, placebo-controlled design, we tested 45 (18 female) healthy volunteers during a baseline session, after application of losartan or placebo (drug session), and during a follow-up session. During each session, participants engaged in a task in which they had to predict the probability of an electrical stimulation on every trial while the true shock contingencies switched repeatedly between phases of high and low shock threat. Computational reinforcement learning models were used to investigate learning dynamics.

RESULTS

Acute administration of losartan significantly reduced participants' adjustment during both low-to-high and high-to-low threat changes. This was driven by reduced aversive learning rates in the losartan group during the drug session compared with baseline. The 50-mg drug dose did not induce reduction of blood pressure or change in reaction times, ruling out a general reduction in attention and engagement. Decreased adjustment of aversive expectations was maintained at a follow-up session 24 hours later.

CONCLUSIONS

This study shows that losartan acutely reduces Pavlovian learning in aversive environments, thereby highlighting a potential role of the renin-angiotensin system in anxiety development.

Naturalistic reinforcement learning

Humans possess a remarkable ability to make decisions within real-world environments that are expansive, complex, and multidimensional. Human cognitive computational neuroscience has sought to exploit reinforcement learning (RL) as a framework within which to explain human decision-making, often focusing on constrained, artificial experimental tasks. In this article, we review recent efforts that use naturalistic approaches to determine how humans make decisions in complex environments that better approximate the real world, providing a clearer picture of how humans navigate the challenges posed by real-world decisions. These studies purposely embed elements of naturalistic complexity within experimental paradigms, rather than focusing on simplification, generating insights into the processes that likely underpin humans' ability to navigate complex, multidimensional real-world environments so successfully.

Depression is associated with reduced outcome sensitivity in a dual valence, magnitude learning task

BACKGROUND

Learning from rewarded and punished choices is perturbed in depressed patients, suggesting that abnormal reinforcement learning may be a cognitive mechanism of the illness. However, previous studies have disagreed about whether this behavior is produced by alterations in the rate of learning or sensitivity to experienced outcomes. This previous work has generally assessed learning in response to binary outcomes of one valence, rather than to both rewarding and punishing continuous outcomes.

METHODS

A novel drifting reward and punishment magnitude reinforcement-learning task was administered to patients with current (n = 40) and remitted depression (n = 39), and healthy volunteers (n = 40) to capture potential differences in learning behavior. Standard questionnaires were administered to measure self-reported depressive symptom severity, trait and state anxiety and level of anhedonic symptoms.

RESULTS

Our findings demonstrate that patients with current depression adjust their learning behaviors to a lesser degree in response to trial-by-trial variations in reward and loss magnitudes than the other groups. Computational modeling revealed that this behavioral signature of current depressive state is better accounted for by reduced reward and punishment sensitivity (all p < 0.031), rather than a change in learning rate (p = 0.708). However, between-group differences were not related to self-reported symptom severity or comorbid anxiety disorders in the current depression group.

CONCLUSION

These findings suggest that current depression is associated with reduced outcome sensitivity rather than altered learning rate. Previous findings reported in this domain mainly from binary learning tasks seem to generalize to learning from continuous outcomes.

Interval Timing as a Computational Pathway From Early Life Adversity to Affective Disorders

Adverse early life experiences can have remarkably enduring negative consequences on mental health, with numerous, varied psychiatric conditions sharing this developmental origin. Yet, the mechanisms linking adverse experiences to these conditions remain poorly understood. Here, we draw on a principled model of interval timing to propose that statistically optimal adaptation of temporal representations to an unpredictable early life environment can produce key characteristics of anhedonia, a transdiagnostic symptom associated with affective disorders like depression and anxiety. The core observation is that early temporal unpredictability produces broader, more imprecise temporal expectations. As a result, reward anticipation is diminished, and associative learning is slowed. When agents with such representations are later introduced to more stable environments, they demonstrate a negativity bias, responding more to the omission of reward than its receipt. Increased encoding of negative events has been proposed to contribute to disorders with anhedonia as a symptom. We then examined how unpredictability interacts with another form of adversity, low reward availability. We found that unpredictability's effect was most strongly felt in richer environments, potentially leading to categorically different phenotypic expressions. In sum, our formalization suggests a single mechanism can help to link early life adversity to a range of behaviors associated with anhedonia, and offers novel insights into the interactive impacts of multiple adversities.

A core component of psychological therapy causes adaptive changes in computational learning mechanisms

BACKGROUND

Cognitive distancing is an emotion regulation strategy commonly used in psychological treatment of various mental health disorders, but its therapeutic mechanisms are unknown.

METHODS

935 participants completed an online reinforcement learning task involving choices between pairs of symbols with differing reward contingencies. Half (49.1%) of the sample was randomised to a cognitive self-distancing intervention and were trained to regulate or 'take a step back' from their emotional response to feedback throughout. Established computational (Q-learning) models were then fit to individuals' choices to derive reinforcement learning parameters capturing clarity of choice values (inverse temperature) and their sensitivity to positive and negative feedback (learning rates).

RESULTS

Cognitive distancing improved task performance, including when participants were later tested on novel combinations of symbols without feedback. Group differences in computational model-derived parameters revealed that cognitive distancing resulted in clearer representations of option values (estimated 0.17 higher inverse temperatures). Simultaneously, distancing caused increased sensitivity to negative feedback (estimated 19% higher loss learning rates). Exploratory analyses suggested this resulted from an evolving shift in strategy by distanced participants: initially, choices were more determined by expected value differences between symbols, but as the task progressed, they became more sensitive to negative feedback, with evidence for a difference strongest by the end of training.

CONCLUSIONS

Adaptive effects on the computations that underlie learning from reward and loss may explain the therapeutic benefits of cognitive distancing. Over time and with practice, cognitive distancing may improve symptoms of mental health disorders by promoting more effective engagement with negative information.

Leveraging Decision Science to Characterize Depression

This brief review examines the potential to use decision science to objectively characterize depression. We provide a brief overview of the existing literature examining different domains of decision-making in depression. Because this overview highlights the specific role of reinforcement learning as an important decision process affected in the disorder, we then introduce reinforcement learning modeling and explain how this approach has identified specific reinforcement learning deficits in depression. We conclude with ideas for future research at the intersection of decision science and depression, emphasizing the potential for decision science to help uncover underlying mechanisms and targets for the treatment of depression.

A Semiparametric Inverse Reinforcement Learning Approach to Characterize Decision Making for Mental Disorders

Major depressive disorder (MDD) is one of the leading causes of disability-adjusted life years. Emerging evidence indicates the presence of reward processing abnormalities in MDD. An important scientific question is whether the abnormalities are due to reduced sensitivity to received rewards or reduced learning ability. Motivated by the probabilistic reward task (PRT) experiment in the EMBARC study, we propose a semiparametric inverse reinforcement learning (RL) approach to characterize the reward-based decision-making of MDD patients. The model assumes that a subject's decision-making process is updated based on a reward prediction error weighted by the subject-specific learning rate. To account for the fact that one favors a decision leading to a potentially high reward, but this decision process is not necessarily linear, we model reward sensitivity with a non-decreasing and nonlinear function. For inference, we estimate the latter via approximation by I-splines and then maximize the joint conditional log-likelihood. We show that the resulting estimators are consistent and asymptotically normal. Through extensive simulation studies, we demonstrate that under different reward-generating distributions, the semiparametric inverse RL outperforms the parametric inverse RL. We apply the proposed method to EMBARC and find that MDD and control groups have similar learning rates but different reward sensitivity functions. There is strong statistical evidence that reward sensitivity functions have nonlinear forms. Using additional brain imaging data in the same study, we find that both reward sensitivity and learning rate are associated with brain activities in the negative affect circuitry under an emotional conflict task.

Approach-avoidance reinforcement learning as a translational and computational model of anxiety-related avoidance

Although avoidance is a prevalent feature of anxiety-related psychopathology, differences in the measurement of avoidance between humans and non-human animals hinder our progress in its theoretical understanding and treatment. To address this, we developed a novel translational measure of anxiety-related avoidance in the form of an approach-avoidance reinforcement learning task, by adapting a paradigm from the non-human animal literature to study the same cognitive processes in human participants. We used computational modelling to probe the putative cognitive mechanisms underlying approach-avoidance behaviour in this task and investigated how they relate to subjective task-induced anxiety. In a large online study (n = 372), participants who experienced greater task-induced anxiety avoided choices associated with punishment, even when this resulted in lower overall reward. Computational modelling revealed that this effect was explained by greater individual sensitivities to punishment relative to rewards. We replicated these findings in an independent sample (n = 627) and we also found fair-to-excellent reliability of measures of task performance in a sub-sample retested 1 week later (n = 57). Our findings demonstrate the potential of approach-avoidance reinforcement learning tasks as translational and computational models of anxiety-related avoidance. Future studies should assess the predictive validity of this approach in clinical samples and experimental manipulations of anxiety.

Anxiety associated with perceived uncontrollable stress enhances expectations of environmental volatility and impairs reward learning

Unavoidable stress can lead to perceived lack of control and learned helplessness, a risk factor for depression. Avoiding punishment and gaining rewards involve updating the values of actions based on experience. Such updating is however useful only if action values are sufficiently stable, something that a lack of control may impair. We examined whether self-reported stress uncontrollability during the first wave of the COVID-19 pandemic predicted impaired reward-learning. In a preregistered study during the first-wave of the COVID-19 pandemic, we used self-reported measures of depression, anxiety, uncontrollable stress, and COVID-19 risk from 427 online participants to predict performance in a three-armed-bandit probabilistic reward learning task. As hypothesised, uncontrollable stress predicted impaired learning, and a greater proportion of probabilistic errors following negative feedback for correct choices, an effect mediated by state anxiety. A parameter from the best-fitting hidden Markov model that estimates expected beliefs that the identity of the optimal choice will shift across images, mediated effects of state anxiety on probabilistic errors and learning deficits. Our findings show that following uncontrollable stress, anxiety promotes an overly volatile representation of the reward-structure of uncertain environments, impairing reward attainment, which is a potential path to anhedonia in depression.

Reduced effects of social feedback on learning in Turner syndrome

Turner syndrome is a genetic condition caused by a complete or partial loss of one of the X chromosomes. Previous studies indicate that Turner syndrome is associated with challenges in social skills, but the underlying mechanisms remain largely unexplored. A possible mechanism is a reduced social influence on learning. The current study examined the impact of social and non-social feedback on learning in women with Turner syndrome (n = 35) and a sex- and age-matched control group (n = 37). Participants were instructed to earn points by repeatedly choosing between two stimuli with unequal probabilities of resulting in a reward. Mastering the task therefore required participants to learn through feedback which of the two stimuli was more likely to be rewarded. Data were analyzed using computational modeling and analyses of choice behavior. Social feedback led to a more explorative choice behavior in the control group, resulting in reduced learning compared to non-social feedback. No effects of social feedback on learning were found in Turner syndrome. The current study thus indicates that women with Turner syndrome may be less sensitive to social influences on reinforcement learning, than the general population.

From Classical Methods to Generative Models: Tackling the Unreliability of Neuroscientific Measures in Mental Health Research

Advances in computational statistics and corresponding shifts in funding initiatives over the past few decades have led to a proliferation of neuroscientific measures being developed in the context of mental health research. Although such measures have undoubtedly deepened our understanding of neural mechanisms underlying cognitive, affective, and behavioral processes associated with various mental health conditions, the clinical utility of such measures remains underwhelming. Recent commentaries point toward the poor reliability of neuroscientific measures to partially explain this lack of clinical translation. Here, we provide a concise theoretical overview of how unreliability impedes clinical translation of neuroscientific measures; discuss how various modeling principles, including those from hierarchical and structural equation modeling frameworks, can help to improve reliability; and demonstrate how to combine principles of hierarchical and structural modeling within the generative modeling framework to achieve more reliable, generalizable measures of brain-behavior relationships for use in mental health research.

The shadowing effect of initial expectation on learning asymmetry

Evidence for positivity and optimism bias abounds in high-level belief updates. However, no consensus has been reached regarding whether learning asymmetries exist in more elementary forms of updates such as reinforcement learning (RL). In RL, the learning asymmetry concerns the sensitivity difference in incorporating positive and negative prediction errors (PE) into value estimation, namely the asymmetry of learning rates associated with positive and negative PEs. Although RL has been established as a canonical framework in characterizing interactions between agent and environment, the direction of learning asymmetry remains controversial. Here, we propose that part of the controversy stems from the fact that people may have different value expectations before entering the learning environment. Such a default value expectation influences how PEs are calculated and consequently biases subjects' choices. We test this hypothesis in two learning experiments with stable or varying reinforcement probabilities, across monetary gains, losses, and gain-loss mixed environments. Our results consistently support the model incorporating both asymmetric learning rates and the initial value expectation, highlighting the role of initial expectation in value updating and choice preference. Further simulation and model parameter recovery analyses confirm the unique contribution of initial value expectation in accessing learning rate asymmetry.

Contextual influence of reinforcement learning performance of depression: evidence for a negativity bias?

BACKGROUNDS

Value-based decision-making impairment in depression is a complex phenomenon: while some studies did find evidence of blunted reward learning and reward-related signals in the brain, others indicate no effect. Here we test whether such reward sensitivity deficits are dependent on the overall value of the decision problem.

METHODS

We used a two-armed bandit task with two different contexts: one 'rich', one 'poor' where both options were associated with an overall positive, negative expected value, respectively. We tested patients (N = 30) undergoing a major depressive episode and age, gender and socio-economically matched controls (N = 26). Learning performance followed by a transfer phase, without feedback, were analyzed to distangle between a decision or a value-update process mechanism. Finally, we used computational model simulation and fitting to link behavioral patterns to learning biases.

RESULTS

Control subjects showed similar learning performance in the 'rich' and the 'poor' contexts, while patients displayed reduced learning in the 'poor' context. Analysis of the transfer phase showed that the context-dependent impairment in patients generalized, suggesting that the effect of depression has to be traced to the outcome encoding. Computational model-based results showed that patients displayed a higher learning rate for negative compared to positive outcomes (the opposite was true in controls).

CONCLUSIONS

Our results illustrate that reinforcement learning performances in depression depend on the value of the context. We show that depressive patients have a specific trouble in contexts with an overall negative state value, which in our task is consistent with a negativity bias at the learning rates level.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Sensitivity and specificity in affective and social learning in adolescence

Adolescence is a period of heightened affective and social sensitivity. In this review we address how this increased sensitivity influences associative learning. Based on recent evidence from human and rodent studies, as well as advances in computational biology, we suggest that, compared to other age groups, adolescents show features of heightened Pavlovian learning but tend to perform worse than adults at instrumental learning. Because Pavlovian learning does not involve decision-making, whereas instrumental learning does, we propose that these developmental differences might be due to heightened sensitivity to rewards and threats in adolescence, coupled with a lower specificity of responding. We discuss the implications of these findings for adolescent mental health and education.

Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty

People radically differ in how they cope with uncertainty. Clinical researchers describe a dispositional characteristic known as "intolerance of uncertainty", a tendency to find uncertainty aversive, reported to be elevated across psychiatric and neurodevelopmental conditions. Concurrently, recent research in computational psychiatry has leveraged theoretical work to characterise individual differences in uncertainty processing. Under this framework, differences in how people estimate different forms of uncertainty can contribute to mental health difficulties. In this review, we briefly outline the concept of intolerance of uncertainty within its clinical context, and we argue that the mechanisms underlying this construct may be further elucidated through modelling how individuals make inferences about uncertainty. We will review the evidence linking psychopathology to different computationally specified forms of uncertainty and consider how these findings might suggest distinct mechanistic routes towards intolerance of uncertainty. We also discuss the implications of this computational approach for behavioural and pharmacological interventions, as well as the importance of different cognitive domains and subjective experiences in studying uncertainty processing.

Spatiotemporal Precision of Neuroimaging in Psychiatry

Aberrant patterns of cognition, perception, and behavior seen in psychiatric disorders are thought to be driven by a complex interplay of neural processes that evolve at a rapid temporal scale. Understanding these dynamic processes in vivo in humans has been hampered by a trade-off between spatial and temporal resolutions inherent to current neuroimaging technology. A recent trend in psychiatric research has been the use of high temporal resolution imaging, particularly magnetoencephalography, often in conjunction with sophisticated machine learning decoding techniques. Developments here promise novel insights into the spatiotemporal dynamics of cognitive phenomena, including domains relevant to psychiatric illnesses such as reward and avoidance learning, memory, and planning. This review considers recent advances afforded by exploiting this increased spatiotemporal precision, with specific reference to applications that seek to drive a mechanistic understanding of psychopathology and the realization of preclinical translation.

Computational Mechanisms of Addiction and Anxiety: A Developmental Perspective

A central goal of computational psychiatry is to identify systematic relationships between transdiagnostic dimensions of psychiatric symptomatology and the latent learning and decision-making computations that inform individuals' thoughts, feelings, and choices. Most psychiatric disorders emerge prior to adulthood, yet little work has extended these computational approaches to study the development of psychopathology. Here, we lay out a roadmap for future studies implementing this approach by developing empirically and theoretically informed hypotheses about how developmental changes in model-based control of action and Pavlovian learning processes may modulate vulnerability to anxiety and addiction. We highlight how insights from studies leveraging computational approaches to characterize the normative developmental trajectories of clinically relevant learning and decision-making processes may suggest promising avenues for future developmental computational psychiatry research.

Belief updating in psychosis, depression and anxiety disorders: A systematic review across computational modelling approaches

Alterations in belief updating are proposed to underpin symptoms of psychiatric illness, including psychosis, depression, and anxiety. Key parameters underlying belief updating can be captured using computational modelling techniques, aiding the identification of unique and shared deficits, and improving diagnosis and treatment. We systematically reviewed research that applied computational modelling to probabilistic tasks measuring belief updating in stable and volatile (changing) environments, across clinical and subclinical psychosis (n = 17), anxiety (n = 9), depression (n = 9) and transdiagnostic samples (n = 9). Depression disorders related to abnormal belief updating in response to the valence of rewards, evidenced in both stable and volatile environments. Whereas psychosis and anxiety disorders were associated with difficulties adapting to changing contingencies specifically, indicating an inflexibility and/or insensitivity to environmental volatility. Higher-order learning models revealed additional difficulties in the estimation of overall environmental volatility across psychosis disorders, showing increased updating to irrelevant information. These findings stress the importance of investigating belief updating in transdiagnostic samples, using homogeneous experimental and computational modelling approaches.

Individual differences in computational psychiatry: a review of current challenges

Bringing precision to the understanding and treatment of mental disorders requires instruments for studying clinically relevant individual differences. One promising approach is the development of computational assays: integrating computational models with cognitive tasks to infer latent patient-specific disease processes in brain computations. While recent years have seen many methodological advancements in computational modelling and many cross-sectional patient studies, much less attention has been paid to basic psychometric properties (reliability and construct validity) of the computational measures provided by the assays. In this review, we assess the extent of this issue by examining emerging empirical evidence. We find that many computational measures suffer from poor psychometric properties, which poses a risk of invalidating previous findings and undermining ongoing research efforts using computational assays to study individual (and even group) differences. We provide recommendations for how to address these problems and, crucially, embed them within a broader perspective on key developments that are needed for translating computational assays to clinical practice.

Stimulating human prefrontal cortex increases reward learning

Work in computational psychiatry suggests that mood disorders may stem from aberrant reinforcement learning processes. Specifically, it has been proposed that depressed individuals believe that negative events are more informative than positive events, resulting in higher learning rates from negative outcomes (Pulcu and Browning, 2019). In this proof-of-concept study, we investigated whether transcranial direct current stimulation (tDCS) applied to dorsolateral prefrontal cortex, as commonly used in depression treatment trials, might change learning rates for affective outcomes. Healthy adults completed an established reinforcement learning task (Pulcu and Browning, 2017) in which the information content of reward and loss outcomes was manipulated by varying the volatility of stimulus-outcome associations. Learning rates on the tasks were quantified using computational models. Stimulation over dorsolateral prefrontal cortex (DLPFC) but not motor cortex (M1) increased learning rates specifically for reward outcomes. The effects of prefrontal tDCS were cognitive state-dependent: tDCS applied during task performance increased learning rates for wins; tDCS applied before task performance decreased both win and loss learning rates. A replication study confirmed the key finding that tDCS to DLPFC during task performance increased learning rates specifically for rewards. Taken together, these findings demonstrate the potential of tDCS for modulating computational parameters of reinforcement learning that are relevant to mood disorders.

Anhedonia and sensitivity to punishment in schizophrenia, depression and opiate use disorder

BACKGROUND

From a behavioural perspective anhedonia is defined as diminished interest in the engagement of pleasurable activities. Despite its presence across a range of psychiatric disorders, the cognitive processes that give rise to anhedonia remain unclear.

METHODS

Here we examine whether anhedonia is associated with learning from positive and negative outcomes in patients diagnosed with major depression, schizophrenia and opiate use disorder alongside a healthy control group. Responses in the Wisconsin Card Sorting Test - a task associated with healthy prefrontal cortex function - were fitted to the Attentional Learning Model (ALM) which separates learning from positive and negative feedback.

RESULTS

Learning from punishment, but not from reward, was negatively associated with anhedonia beyond other socio-demographic, cognitive and clinical variables. This impairment in punishment sensitivity was also associated with faster responses following negative feedback, independently of the degree of surprise.

LIMITATIONS

Future studies should test the longitudinal association between punishment sensitivity and anhedonia also in other clinical populations controlling for the effect of specific medications.

CONCLUSIONS

Together the results reveal that anhedonic subjects, because of their negative expectations, are less sensitive to negative feedbacks; this might lead them to persist in actions leading to negative outcomes.

Formalising social representation to explain psychiatric symptoms

Recent work in social cognition has moved beyond a focus on how people process social rewards to examine how healthy people represent other agents and how this is altered in psychiatric disorders. However, formal modelling of social representation has not kept pace with these changes, impeding our understanding of how core aspects of social cognition function, and fail, in psychopathology. Here, we suggest that belief-based computational models provide a basis for an integrated sociocognitive approach to psychiatry, with the potential to address important but unexamined pathologies of social representation, such as maladaptive schemas and illusory social agents.

The computational psychiatry of antisocial behaviour and psychopathy

Antisocial behaviours such as disobedience, lying, stealing, destruction of property, and aggression towards others are common to multiple disorders of childhood and adulthood, including conduct disorder, oppositional defiant disorder, psychopathy, and antisocial personality disorder. These disorders have a significant negative impact for individuals and for society, but whether they represent clinically different phenomena, or simply different approaches to diagnosing the same underlying psychopathology is highly debated. Computational psychiatry, with its dual focus on identifying different classes of disorder and health (data-driven) and latent cognitive and neurobiological mechanisms (theory-driven), is well placed to address these questions. The elucidation of mechanisms that might characterise latent processes across different disorders of antisocial behaviour can also provide important advances. In this review, we critically discuss the contribution of computational research to our understanding of various antisocial behaviour disorders, and highlight suggestions for how computational psychiatry can address important clinical and scientific questions about these disorders in the future.

Computational models of subjective feelings in psychiatry

Research in computational psychiatry is dominated by models of behavior. Subjective experience during behavioral tasks is not well understood, even though it should be relevant to understanding the symptoms of psychiatric disorders. Here, we bridge this gap and review recent progress in computational models for subjective feelings. For example, happiness reflects not how well people are doing, but whether they are doing better than expected. This dependence on recent reward prediction errors is intact in major depression, although depressive symptoms lower happiness during tasks. Uncertainty predicts subjective feelings of stress in volatile environments. Social prediction errors influence feelings of self-worth more in individuals with low self-esteem despite a reduced willingness to change beliefs due to social feedback. Measuring affective state during behavioral tasks provides a tool for understanding psychiatric symptoms that can be dissociable from behavior. When smartphone tasks are collected longitudinally, subjective feelings provide a potential means to bridge the gap between lab-based behavioral tasks and real-life behavior, emotion, and psychiatric symptoms.

Reinforcement-based responsiveness, depression, and anhedonia: A multi-method investigation of intergenerational risk

Offspring of depressed parents are at an increased risk for depression. Reward- and punishment-based systems might be mechanisms linking maternal outcomes to offspring depression and anhedonia. The current study was designed to investigate the intergenerational relations between maternal markers of reward and punishment responsiveness and their offspring's depression and anhedonia in a community sample of 40 mother (mean age = 44.5; SD = 6.82) and adolescent (mean age = 14.73; SD = 1.25; 52.5% female) dyads. Maternal markers of reward and punishment responsiveness were captured using self-report, behavioral, and neurophysiological methods, and self-reported depression and anhedonia symptoms were used as outcomes among the adolescent offspring. Maternal self-reported reward responsiveness and punishment learning rates were differentially associated with depression across male and female offspring. Regarding anhedonia, maternal punishment learning rate was positively related to adolescent anhedonia regardless of offspring biological sex. Maternal reward learning rate was also positively associated with anhedonia among male offspring. In general, low concurrence across self-report, behavioral, and neurophysiological markers of reward and punishment responsiveness was found. The results from the current study suggest that learning-rates on reinforcement-based behavioral tasks may be important objective markers to consider when evaluating intergenerational risk.

Individual differences in naturalistic learning link negative emotionality to the development of anxiety

Organisms learn from prediction errors (PEs) to predict the future. Laboratory studies using small financial outcomes find that humans use PEs to update expectations and link individual differences in PE-based learning to internalizing disorders. Because of the low-stakes outcomes in most tasks, it is unclear whether PE learning emerges in naturalistic, high-stakes contexts and whether individual differences in PE learning predict psychopathology risk. Using experience sampling to assess 625 college students' expected exam grades, we found evidence of PE-based learning and a general tendency to discount negative PEs, an "optimism bias." However, individuals with elevated negative emotionality, a personality trait linked to the development of anxiety disorders, displayed a global pessimism and learning differences that impeded accurate expectations and predicted future anxiety symptoms. A sensitivity to PEs combined with an aversion to negative PEs may result in a pessimistic and inaccurate model of the world, leading to anxiety.

Social feedback enhances learning in Williams syndrome

Williams syndrome (WS) is a rare genetic condition characterized by high social interest and approach motivation as well as intellectual disability and anxiety. Despite the fact that social stimuli are believed to have an increased intrinsic reward value in WS, it is not known whether this translates to learning and decision making. Genes homozygously deleted in WS are linked to sociability in the general population, making it a potential model condition for understanding the social brain. Probabilistic reinforcement learning was studied with either social or non-social rewards for correct choices. Social feedback improved learning in individuals with Williams syndrome but not in typically developing controls or individuals with other intellectual disabilities. Computational modeling indicated that these effects on social feedback were mediated by a shift towards higher weight given to rewards relative to punishments and increased choice consistency. We conclude that reward learning in WS is characterized by high volatility and a tendency to learn how to avoid punishment rather than how to gain rewards. Social feedback can partly normalize this pattern and promote adaptive reward learning.

Computational perspectives on human fear and anxiety

Fear and anxiety are adaptive emotions that serve important defensive functions, yet in excess, they can be debilitating and lead to poor mental health. Computational modelling of behaviour provides a mechanistic framework for understanding the cognitive and neurobiological bases of fear and anxiety, and has seen increasing interest in the field. In this brief review, we discuss recent developments in the computational modelling of human fear and anxiety. Firstly, we describe various reinforcement learning strategies that humans employ when learning to predict or avoid threat, and how these relate to symptoms of fear and anxiety. Secondly, we discuss initial efforts to explore, through a computational lens, approach-avoidance conflict paradigms that are popular in animal research to measure fear- and anxiety-relevant behaviours. Finally, we discuss negative biases in decision-making in the face of uncertainty in anxiety.

The naturalistic reinforcement of worry from positive and negative emotional contrasts: Results from a momentary assessment study within social interactions

BACKGROUND

The Contrast Avoidance Model (Newman & Llera, 2011) proposes that worry is reinforced by avoiding a negative contrast and increasing the likelihood of a positive contrast.

OBJECTIVE

To determine if reinforcement of worry occurs naturalistically via contrasts in both negative and positive emotion.

METHOD

Using event-contingent momentary assessment we assessed social interactions, pre-interaction state worry and pre-post interaction positive and negative emotion. Participants with generalized anxiety disorder (GAD; N = 83) completed an online questionnaire after social interactions lasting at least 1 min for 8 days. Three-level multilevel models were conducted.

RESULTS

Higher worry was concurrently associated with increased negative emotion and decreased positive emotion. Regardless of pre-interaction worry level, negative emotion decreased, and positive emotion increased from before to after interactions, suggesting that most interactions were benign or positive. At lower levels of pre-interaction worry, participants experienced increased negative emotion and decreased positive emotion from before to after interactions. At higher levels of pre-interaction worry, participants experienced decreased negative emotion and increased positive emotion from before to after interactions.

CONCLUSION

Among persons with GAD, worrying before social interactions may be both negatively and positively reinforced; furthermore, not worrying before social interactions may be both negatively and positively punished.

Cardiac Cycle Affects the Asymmetric Value Updating in Instrumental Reward Learning

This study aimed to investigate whether instrumental reward learning is affected by the cardiac cycle. To this end, we examined the effects of the cardiac cycle (systole or diastole) on the computational processes underlying the participants’ choices in the instrumental learning task. In the instrumental learning task, participants were required to select one of two discriminative stimuli (neutral visual stimuli) and immediately receive reward/punishment feedback depending on the probability assigned to the chosen stimuli. To manipulate the cardiac cycle, the presentation of discriminative stimuli was timed to coincide with either cardiac systole or diastole. We fitted the participants’ choices in the task with reinforcement learning (RL) models and estimated parameters involving instrumental learning (i.e., learning rate and inverse temperature) separately in the systole and diastole trials. Model-based analysis revealed that the learning rate for positive prediction errors was higher than that for negative prediction errors in the systole trials; however, learning rates did not differ between positive and negative prediction errors in the diastole trials. These results demonstrate that the natural fluctuation of cardiac afferent signals can affect asymmetric value updating in instrumental reward learning.