Reinforcement learning in depression: A review of computational research

Electrical brain activations in preadolescents during a probabilistic reward-learning task reflect cognitive processes and behavioral strategy

Both adults and children learn through feedback which environmental events and choices are associated with higher probability of reward, an ability thought to be supported by the development of fronto-striatal reward circuits. Recent developmental studies have applied computational models of reward learning to investigate such learning in children. However, tasks and measures effective for assaying the cascade of reward-learning neural processes in children have been limited. Using a child-version of a probabilistic reward-learning task while recording event-related-potential (ERP) measures of electrical brain activity, this study examined key processes of reward learning in preadolescents (8-12 years old; n=30), namely: (1) reward-feedback sensitivity, as measured by the early-latency, reward-related, frontal ERP positivity, (2) rapid attentional shifting of processing toward favored visual stimuli, as measured by the N2pc component, and (3) longer-latency attention-related responses to reward feedback as a function of behavioral strategies (i.e., Win-Stay-Lose-Shift), as measured by the central-parietal P300. Consistent with our prior work in adults, the behavioral findings indicate preadolescents can learn stimulus-reward outcome associations, but at varying levels of performance. Neurally, poor preadolescent learners (those with slower learning rates) showed greater reward-related positivity amplitudes relative to good learners, suggesting greater reward-feedback sensitivity. We also found attention shifting towards to-be-chosen stimuli, as evidenced by the N2pc, but not to more highly rewarded stimuli as we have observed in adults. Lastly, we found the behavioral learning strategy (i.e., Win-Stay-Lose-Shift) reflected by the feedback-elicited parietal P300. These findings provide novel insights into the key neural processes underlying reinforcement learning in preadolescents.

Artificial neural networks for model identification and parameter estimation in computational cognitive models

Computational cognitive models have been used extensively to formalize cognitive processes. Model parameters offer a simple way to quantify individual differences in how humans process information. Similarly, model comparison allows researchers to identify which theories, embedded in different models, provide the best accounts of the data. Cognitive modeling uses statistical tools to quantitatively relate models to data that often rely on computing/estimating the likelihood of the data under the model. However, this likelihood is computationally intractable for a substantial number of models. These relevant models may embody reasonable theories of cognition, but are often under-explored due to the limited range of tools available to relate them to data. We contribute to filling this gap in a simple way using artificial neural networks (ANNs) to map data directly onto model identity and parameters, bypassing the likelihood estimation. We test our instantiation of an ANN as a cognitive model fitting tool on classes of cognitive models with strong inter-trial dependencies (such as reinforcement learning models), which offer unique challenges to most methods. We show that we can adequately perform both parameter estimation and model identification using our ANN approach, including for models that cannot be fit using traditional likelihood-based methods. We further discuss our work in the context of the ongoing research leveraging simulation-based approaches to parameter estimation and model identification, and how these approaches broaden the class of cognitive models researchers can quantitatively investigate.

Early-Life Adversities Are Associated With Lower Expected Value Signaling in the Adult Brain

BACKGROUND

Early adverse experiences are assumed to affect fundamental processes of reward learning and decision making. However, computational neuroimaging studies investigating these circuits in the context of adversity are sparse and limited to studies conducted in adolescent samples, leaving the long-term effects unexplored.

METHODS

Using data from a longitudinal birth cohort study (n = 156; 87 female), we investigated associations between adversities and computational markers of reward learning (i.e., expected value, prediction errors). At age 33 years, all participants completed a functional magnetic resonance imaging-based passive avoidance task. Psychopathology measures were collected at the time of functional magnetic resonance imaging investigation and during the COVID-19 pandemic. We applied a principal component analysis to capture common variations across 7 adversity measures. The resulting adversity factors (factor 1: postnatal psychosocial adversities and prenatal maternal smoking; factor 2: prenatal maternal stress and obstetric adversity; factor 3: lower maternal stimulation) were linked with psychopathology and neural responses in the core reward network using multiple regression analysis.

RESULTS

We found that the adversity dimension primarily informed by lower maternal stimulation was linked to lower expected value representation in the right putamen, right nucleus accumbens, and anterior cingulate cortex. Expected value encoding in the right nucleus accumbens further mediated the relationship between this adversity dimension and psychopathology and predicted higher withdrawn symptoms during the COVID-19 pandemic.

CONCLUSIONS

Our results suggested that early adverse experiences in caregiver context might have a long-term disruptive effect on reward learning in reward-related brain regions, which can be associated with suboptimal decision making and thereby may increase the vulnerability of developing psychopathology.

Artificial neural networks for model identification and parameter estimation in computational cognitive models

Computational cognitive models have been used extensively to formalize cognitive processes. Model parameters offer a simple way to quantify individual differences in how humans process information. Similarly, model comparison allows researchers to identify which theories, embedded in different models, provide the best accounts of the data. Cognitive modeling uses statistical tools to quantitatively relate models to data that often rely on computing/estimating the likelihood of the data under the model. However, this likelihood is computationally intractable for a substantial number of models. These relevant models may embody reasonable theories of cognition, but are often under-explored due to the limited range of tools available to relate them to data. We contribute to filling this gap in a simple way using artificial neural networks (ANNs) to map data directly onto model identity and parameters, bypassing the likelihood estimation. We test our instantiation of an ANN as a cognitive model fitting tool on classes of cognitive models with strong inter-trial dependencies (such as reinforcement learning models), which offer unique challenges to most methods. We show that we can adequately perform both parameter estimation and model identification using our ANN approach, including for models that cannot be fit using traditional likelihood-based methods. We further discuss our work in the context of the ongoing research leveraging simulation-based approaches to parameter estimation and model identification, and how these approaches broaden the class of cognitive models researchers can quantitatively investigate.

Ketamine produces no detectable long-term positive or negative effects on cognitive flexibility or reinforcement learning of male rats

RATIONALE

Patients with major depressive disorder (MDD) often experience abnormalities in behavioral adaptation following environmental changes (i.e., cognitive flexibility) and tend to undervalue positive outcomes but overvalue negative outcomes. The probabilistic reversal learning task (PRL) is used to study these deficits across species and to explore drugs that may have therapeutic value. Selective serotonin-reuptake inhibitors (SSRIs) have limited effectiveness in treating MDD and produce inconsistent effects in non-human versions of the PRL. As such, ketamine, a novel and potentially rapid-acting therapeutic, has begun to be examined using the PRL. Two previous studies examining the effects of ketamine in the PRL have shown conflicting results and only examined short-term effects of ketamine.

OBJECTIVE

This experiment examined PRL performance across a 2-week period following a single exposure to a ketamine dose that varied across groups.

METHODS

After five sessions of PRL training, groups of rats received an injection of either 0, 10, 20 or 30 mg/kg ketamine. One-hour post-injection, rats engaged in the PRL, and subsequently sessions continued daily for 2 weeks. Traditional behavioral and computational reinforcement learning-derived measures were examined.

RESULTS

Results showed that ketamine had acute effects 1-h post-injection, including a significant decrease in the value of the punishment learning rate. Beyond 1 h, ketamine produced no detectable improvements nor decrements in performance across 2 weeks.

CONCLUSION

Overall, the present results suggest that the range of ketamine doses examined do not have long-term positive or negative effects on cognitive flexibility or reward processing in healthy rats as measured by the PRL.

Perceptual Observer Modeling Reveals Likely Mechanisms of Face Expression Recognition Deficits in Depression

BACKGROUND

Deficits in face emotion recognition are well documented in depression, but the underlying mechanisms are poorly understood. Psychophysical observer models provide a way to precisely characterize such mechanisms. Using model-based analyses, we tested 2 hypotheses about how depression may reduce sensitivity to detect face emotion: 1) via a change in selectivity for visual information diagnostic of emotion or 2) via a change in signal-to-noise ratio in the system performing emotion detection.

METHODS

Sixty adults, one half meeting criteria for major depressive disorder and the other half healthy control participants, identified sadness and happiness in noisy face stimuli, and their responses were used to estimate templates encoding the visual information used for emotion identification. We analyzed these templates using traditional and model-based analyses; in the latter, the match between templates and stimuli, representing sensory evidence for the information encoded in the template, was compared against behavioral data.

RESULTS

Estimated happiness templates produced sensory evidence that was less strongly correlated with response times in participants with depression than in control participants, suggesting that depression was associated with a reduced signal-to-noise ratio in the detection of happiness. The opposite results were found for the detection of sadness. We found little evidence that depression was accompanied by changes in selectivity (i.e., information used to detect emotion), but depression was associated with a stronger influence of face identity on selectivity.

CONCLUSIONS

Depression is more strongly associated with changes in signal-to-noise ratio during emotion recognition, suggesting that deficits in emotion detection are driven primarily by deprecated signal quality rather than suboptimal sampling of information used to detect emotion.

Transdiagnostic inflexible learning dynamics explain deficits in depression and schizophrenia

Deficits in reward learning are core symptoms across many mental disorders. Recent work suggests that such learning impairments arise by a diminished ability to use reward history to guide behaviour, but the neuro-computational mechanisms through which these impairments emerge remain unclear. Moreover, limited work has taken a transdiagnostic approach to investigate whether the psychological and neural mechanisms that give rise to learning deficits are shared across forms of psychopathology. To provide insight into this issue, we explored probabilistic reward learning in patients diagnosed with major depressive disorder (n = 33) or schizophrenia (n = 24) and 33 matched healthy controls by combining computational modelling and single-trial EEG regression. In our task, participants had to integrate the reward history of a stimulus to decide whether it is worthwhile to gamble on it. Adaptive learning in this task is achieved through dynamic learning rates that are maximal on the first encounters with a given stimulus and decay with increasing stimulus repetitions. Hence, over the course of learning, choice preferences would ideally stabilize and be less susceptible to misleading information. We show evidence of reduced learning dynamics, whereby both patient groups demonstrated hypersensitive learning (i.e. less decaying learning rates), rendering their choices more susceptible to misleading feedback. Moreover, there was a schizophrenia-specific approach bias and a depression-specific heightened sensitivity to disconfirmational feedback (factual losses and counterfactual wins). The inflexible learning in both patient groups was accompanied by altered neural processing, including no tracking of expected values in either patient group. Taken together, our results thus provide evidence that reduced trial-by-trial learning dynamics reflect a convergent deficit across depression and schizophrenia. Moreover, we identified disorder distinct learning deficits.

The association of positive emotions with absenteeism and presenteeism in Japanese workers

BACKGROUND

Negative emotions such as depression have been associated with increased absenteeism and presenteeism, contributing to substantial economic loss. However, no study has investigated if positive emotions such as happiness influence absenteeism and presenteeism.

METHODS

Using data from the Japan COVID-19 and Society Internet Survey (JACSIS), a nationwide survey conducted in September-October 2022 (n = 19,214), we investigated if two major, representative positive emotions (happiness and gratitude) are associated with absenteeism and presenteeism. Absenteeism was defined as reporting more than one day of sick leave in the past one month. Presenteeism was measured with the Work Functioning Impairment Scale. Logistic regression was used to estimate odds ratios.

RESULTS

12.4 % and 21.8 % of subjects reported absenteeism and presenteeism, respectively. Logistic regression estimated that after adjusting covariates, happiness was associated with lower odds of absenteeism (OR = 0.792, 95 % CI [0.706, 0.888]) and presenteeism (OR = 0.531, 95 % CI [0.479, 0.588]) while gratitude was associated with lower odds of presenteeism only (OR = 0.705, 95 % CI [0.643, 0.774]). Furthermore, simultaneous presence of both happiness and gratitude was associated with further lower odds of presenteeism (OR = 0.385, 95%CI [0.338, 0.439]), indicating a synergetic relation.

DISCUSSION

This study is the first to investigate the association between positive emotions and absenteeism and presenteeism. Given the substantial economic loss due to absenteeism and presenteeism, strategies to enhance positive emotions are necessary.

Pramipexole for the Treatment of Depression: Efficacy and Mechanisms

Dopaminergic mechanisms are a plausible treatment target for patients with clinical depression but are relatively underexplored in conventional antidepressant medications. There is continuing interest in the potential antidepressant effects of the dopamine receptor agonist, pramipexole, with data from both case series and controlled trials indicating that this agent may produce benefit for patients with difficult-to-treat depression. Pramipexole's therapeutic utility in depression is likely to be expressed through alterations in reward mechanisms which are strongly influenced by dopamine pathways and are known to function abnormally in depressed patients. Our work in healthy participants using brain imaging in conjunction with computational modelling suggests that repeated pramipexole facilitates reward learning by inhibiting value decay. This mechanism needs to be confirmed in larger clinical trials in depressed patients. Such studies will also allow assessment of whether baseline performance in reward learning in depression predicts therapeutic response to pramipexole treatment.

Reinforcement learning and working memory in mood disorders: A computational analysis in a developmental transdiagnostic sample

BACKGROUND

Mood disorders commonly onset during adolescence and young adulthood and are conceptually and empirically related to reinforcement learning abnormalities. However, the nature of abnormalities associated with acute symptom severity versus lifetime diagnosis remains unclear, and prior research has often failed to disentangle working memory from reward processes.

METHODS

The present sample (N = 220) included adolescents and young adults with a lifetime history of unipolar disorders (n = 127), bipolar disorders (n = 28), or no history of psychopathology (n = 62), and varying severity of mood symptoms. Analyses fitted a reinforcement learning and working memory model to an instrumental learning task that varied working memory load, and tested associations between model parameters and diagnoses or current symptoms.

RESULTS

Current severity of manic or anhedonic symptoms negatively correlated with task performance. Participants reporting higher severity of current anhedonia, or with lifetime unipolar or bipolar disorders, showed lower reward learning rates. Participants reporting higher severity of current manic symptoms showed faster working memory decay and reduced use of working memory.

LIMITATIONS

Computational parameters should be interpreted in the task environment (a deterministic reward learning paradigm), and developmental population. Future work should test replication in other paradigms and populations.

CONCLUSIONS

Results indicate abnormalities in reinforcement learning processes that either scale with current symptom severity, or correspond with lifetime mood diagnoses. Findings may have implications for understanding reward processing anomalies related to state-like (current symptom) or trait-like (lifetime diagnosis) aspects of mood disorders.

Anxiety associated with perceived uncontrollable stress enhances expectations of environmental volatility and impairs reward learning

Unavoidable stress can lead to perceived lack of control and learned helplessness, a risk factor for depression. Avoiding punishment and gaining rewards involve updating the values of actions based on experience. Such updating is however useful only if action values are sufficiently stable, something that a lack of control may impair. We examined whether self-reported stress uncontrollability during the first wave of the COVID-19 pandemic predicted impaired reward-learning. In a preregistered study during the first-wave of the COVID-19 pandemic, we used self-reported measures of depression, anxiety, uncontrollable stress, and COVID-19 risk from 427 online participants to predict performance in a three-armed-bandit probabilistic reward learning task. As hypothesised, uncontrollable stress predicted impaired learning, and a greater proportion of probabilistic errors following negative feedback for correct choices, an effect mediated by state anxiety. A parameter from the best-fitting hidden Markov model that estimates expected beliefs that the identity of the optimal choice will shift across images, mediated effects of state anxiety on probabilistic errors and learning deficits. Our findings show that following uncontrollable stress, anxiety promotes an overly volatile representation of the reward-structure of uncertain environments, impairing reward attainment, which is a potential path to anhedonia in depression.

Signed and unsigned effects of prediction error on memory: Is it a matter of choice?

Adaptive decision-making is governed by at least two types of memory processes. On the one hand, learned predictions through integrating multiple experiences, and on the other hand, one-shot episodic memories. These two processes interact, and predictions - particularly prediction errors - influence how episodic memories are encoded. However, studies using computational models disagree on the exact shape of this relationship, with some findings showing an effect of signed prediction errors and others showing an effect of unsigned prediction errors on episodic memory. We argue that the choice-confirmation bias, which reflects stronger learning from choice-confirming compared to disconfirming outcomes, could explain these seemingly diverging results. Our perspective implies that the influence of prediction errors on episodic encoding critically depends on whether people can freely choose between options (i.e., instrumental learning tasks) or not (Pavlovian learning tasks). The choice-confirmation bias on memory encoding might have evolved to prioritize memory representations that optimize reward-guided decision-making. We conclude by discussing open issues and implications for future studies.

Toward a computational understanding of how reminiscing about positive autobiographical memories influences decision-making under risk

Recent computational psychiatric research has dissected decision-making under risk into different underlying cognitive computational constructs and identified disease-specific changes in these constructs. Studies are underway to investigate what kind of behavioral or psychological interventions can restore these cognitive, computational constructs. In our previous study, we showed that reminiscing about positive autobiographical memories reduced risk aversion and affected probability weighting in the opposite direction from that observed in psychiatric disorders. However, in that study, we compared positive versus neutral memory retrieval by using a within-subjects crossover posttest design. Therefore, the change of decision-making from baseline is unclear. Furthermore, we used a hypothetical decision-making task and did not include monetary incentives. We attempt to address these limitations and investigated how reminiscing about positive autobiographical memories influences decision-making under risk using a between-subjects pretest posttest comparison design with performance-contingent monetary incentives. In thirty-eight healthy, young adults, we found that reminiscing about positive memories reinforced the commonly observed inverted S-shaped nonlinear probability weighting (f = 0.345, medium to large in effect size). In contrast, reminiscing about positive memories did not affect risk aversion in general. Given that the change in probability weighting after reminiscing about positive memories is in the opposite direction from that observed in psychiatric disorders, our results indicate that positive autobiographical memory retrieval might be a useful behavioral intervention strategy for amending the altered decision-making under risk in psychiatric diseases.

Reliability of gamified reinforcement learning in densely sampled longitudinal assessments

Reinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. In this paper, we present an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) of current mental and physiological states for repeated assessment over weeks. In this task, players have to identify the most effective medication by integrating reward values with changing probabilities to win (according to random Gaussian walks). Participants can complete up to 31 runs with 150 trials each. To encourage replay, in-game screens provide feedback on the progress. Using an initial validation sample of 384 players (9729 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show poor to fair intra-class correlations (ICC: 0.22-0.53), indicating substantial within- and between-subject variance. Notably, items assessing the psychological state showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

Adaptive Design Optimization as a Promising Tool for Reliable and Efficient Computational Fingerprinting

A key challenge in understanding mental (dys)functions is their etiological and functional heterogeneity, and several multidimensional assessments have been proposed for their comprehensive characterization. However, such assessments require lengthy testing, which may hinder reliable and efficient characterization of individual differences due to increased fatigue and distraction, especially in clinical populations. Computational modeling may address this challenge as it often provides more reliable measures of latent neurocognitive processes underlying observed behaviors and captures individual differences better than traditional assessments. However, even with a state-of-the-art hierarchical modeling approach, reliable estimation of model parameters still requires a large number of trials. Recent work suggests that Bayesian adaptive design optimization (ADO) is a promising way to address these challenges. With ADO, experimental design is optimized adaptively from trial to trial to extract the maximum amount of information about an individual's characteristics. In this review, we first describe the ADO methodology and then summarize recent work demonstrating that ADO increases the reliability and efficiency of latent neurocognitive measures. We conclude by discussing the challenges and future directions of ADO and proposing development of ADO-based computational fingerprints to reliably and efficiently characterize the heterogeneous profiles of psychiatric disorders.

Contextual influence of reinforcement learning performance of depression: evidence for a negativity bias?

BACKGROUNDS

Value-based decision-making impairment in depression is a complex phenomenon: while some studies did find evidence of blunted reward learning and reward-related signals in the brain, others indicate no effect. Here we test whether such reward sensitivity deficits are dependent on the overall value of the decision problem.

METHODS

We used a two-armed bandit task with two different contexts: one 'rich', one 'poor' where both options were associated with an overall positive, negative expected value, respectively. We tested patients (N = 30) undergoing a major depressive episode and age, gender and socio-economically matched controls (N = 26). Learning performance followed by a transfer phase, without feedback, were analyzed to distangle between a decision or a value-update process mechanism. Finally, we used computational model simulation and fitting to link behavioral patterns to learning biases.

RESULTS

Control subjects showed similar learning performance in the 'rich' and the 'poor' contexts, while patients displayed reduced learning in the 'poor' context. Analysis of the transfer phase showed that the context-dependent impairment in patients generalized, suggesting that the effect of depression has to be traced to the outcome encoding. Computational model-based results showed that patients displayed a higher learning rate for negative compared to positive outcomes (the opposite was true in controls).

CONCLUSIONS

Our results illustrate that reinforcement learning performances in depression depend on the value of the context. We show that depressive patients have a specific trouble in contexts with an overall negative state value, which in our task is consistent with a negativity bias at the learning rates level.

Biased belief updating in depression

Cognitive approaches to depression have benefitted from recent research on belief updating, examining how new information is used to alter beliefs. This review presents recent advances in understanding various sources of bias in belief updating in depression. Specifically, research has demonstrated that people with depression have difficulty revising negative beliefs in response to novel positive information, whereas belief updating in depression is not related to an enhanced integration of negative information. In terms of mechanisms underlying the deficient processing of positive information, research has shown that people with depression use defensive cognitive strategies to devalue novel positive information. Furthermore, the disregard of novel positive information can be amplified by the presence of state negative affect, and the resulting persistence of negative beliefs in turn perpetuates chronically low mood, contributing to a self-reinforcing negative feedback loop of beliefs and affect. Synthesising previous research, this review proposes a coherent framework of when belief change is likely to occur, and argues that future research also needs to elucidate why people with depression hesitate to abandon negative beliefs. Recent insights from belief updating have not only improved the understanding of the psychopathology of depression, but also have the potential to improve its cognitive-behavioural treatment.

Spatiotemporal Precision of Neuroimaging in Psychiatry

Aberrant patterns of cognition, perception, and behavior seen in psychiatric disorders are thought to be driven by a complex interplay of neural processes that evolve at a rapid temporal scale. Understanding these dynamic processes in vivo in humans has been hampered by a trade-off between spatial and temporal resolutions inherent to current neuroimaging technology. A recent trend in psychiatric research has been the use of high temporal resolution imaging, particularly magnetoencephalography, often in conjunction with sophisticated machine learning decoding techniques. Developments here promise novel insights into the spatiotemporal dynamics of cognitive phenomena, including domains relevant to psychiatric illnesses such as reward and avoidance learning, memory, and planning. This review considers recent advances afforded by exploiting this increased spatiotemporal precision, with specific reference to applications that seek to drive a mechanistic understanding of psychopathology and the realization of preclinical translation.

Invasive Computational Psychiatry

Computational psychiatry, a relatively new yet prolific field that aims to understand psychiatric disorders with formal theories about the brain, has seen tremendous growth in the past decade. Despite initial excitement, actual progress made by computational psychiatry seems stagnant. Meanwhile, understanding of the human brain has benefited tremendously from recent progress in intracranial neuroscience. Specifically, invasive techniques such as stereotactic electroencephalography, electrocorticography, and deep brain stimulation have provided a unique opportunity to precisely measure and causally modulate neurophysiological activity in the living human brain. In this review, we summarize progress and drawbacks in both computational psychiatry and invasive electrophysiology and propose that their combination presents a highly promising new direction-invasive computational psychiatry. The value of this approach is at least twofold. First, it advances our mechanistic understanding of the neural computations of mental states by providing a spatiotemporally precise depiction of neural activity that is traditionally unattainable using noninvasive techniques with human subjects. Second, it offers a direct and immediate way to modulate brain states through stimulation of algorithmically defined neural regions and circuits (i.e., algorithmic targeting), thus providing both causal and therapeutic insights. We then present depression as a use case where the combination of computational and invasive approaches has already shown initial success. We conclude by outlining future directions as a road map for this exciting new field as well as presenting cautions about issues such as ethical concerns and generalizability of findings.

Physical activity for cognitive health promotion: An overview of the underlying neurobiological mechanisms

Physical activity is one of the modifiable factors of cognitive decline and dementia with the strongest evidence. Although many influential reviews have illustrated the neurobiological mechanisms of the cognitive benefits of physical activity, none of them have linked the neurobiological mechanisms to normal exercise physiology to help the readers gain a more advanced, comprehensive understanding of the phenomenon. In this review, we address this issue and provide a synthesis of the literature by focusing on five most studied neurobiological mechanisms. We show that the body's adaptations to enhance exercise performance also benefit the brain and contribute to improved cognition. Specifically, these adaptations include, 1), the release of growth factors that are essential for the development and growth of neurons and for neurogenesis and angiogenesis, 2), the production of lactate that provides energy to the brain and is involved in the synthesis of glutamate and the maintenance of long-term potentiation, 3), the release of anti-inflammatory cytokines that reduce neuroinflammation, 4), the increase in mitochondrial biogenesis and antioxidant enzyme activity that reduce oxidative stress, and 5), the release of neurotransmitters such as dopamine and 5-HT that regulate neurogenesis and modulate cognition. We also discussed several issues relevant for prescribing physical activity, including what intensity and mode of physical activity brings the most cognitive benefits, based on their influence on the above five neurobiological mechanisms. We hope this review helps readers gain a general understanding of the state-of-the-art knowledge on the neurobiological mechanisms of the cognitive benefits of physical activity and guide them in designing new studies to further advance the field.

Formalising social representation to explain psychiatric symptoms

Recent work in social cognition has moved beyond a focus on how people process social rewards to examine how healthy people represent other agents and how this is altered in psychiatric disorders. However, formal modelling of social representation has not kept pace with these changes, impeding our understanding of how core aspects of social cognition function, and fail, in psychopathology. Here, we suggest that belief-based computational models provide a basis for an integrated sociocognitive approach to psychiatry, with the potential to address important but unexamined pathologies of social representation, such as maladaptive schemas and illusory social agents.

The impact of ADHD symptom severity on reinforcement and punishment learning among adults

Introduction: Aberrations in feedback learning are hypothesised to contribute to the behavioural disruptions and impairment of attention-deficit/hyperactivity disorder (ADHD). However, few studies have evaluated the relation of reward/punishment feedback and ADHD symptom severity on learning. The current study evaluates the differential effects of reward and punishment feedback on learning among adults with elevated ADHD. Methods: One hundred five participants self-reported their level of current ADHD symptoms and completed an innovative instrumental learning task. Results: Consistent with predictions, participants with low self-reported ADHD symptom severity benefitted equally from reward and punishment feedback during the learning task, whereas participants with high self-reported symptom severity performed better (indexed by accuracy on learning task) from reward than punishment feedback trials. Conclusions: Overall, adults with high self-reported symptom severity of ADHD learned more from reward-based feedback, which provides critical implications for motivational theories about ADHD, as well as for treatment protocols. Future work should examine the translatability of results within a treatment setting.

Angiotensin blockade enhances motivational reward learning via enhancing striatal prediction error signaling and frontostriatal communication

Adaptive human learning utilizes reward prediction errors (RPEs) that scale the differences between expected and actual outcomes to optimize future choices. Depression has been linked with biased RPE signaling and an exaggerated impact of negative outcomes on learning which may promote amotivation and anhedonia. The present proof-of-concept study combined computational modeling and multivariate decoding with neuroimaging to determine the influence of the selective competitive angiotensin II type 1 receptor antagonist losartan on learning from positive or negative outcomes and the underlying neural mechanisms in healthy humans. In a double-blind, between-subjects, placebo-controlled pharmaco-fMRI experiment, 61 healthy male participants (losartan, n = 30; placebo, n = 31) underwent a probabilistic selection reinforcement learning task incorporating a learning and transfer phase. Losartan improved choice accuracy for the hardest stimulus pair via increasing expected value sensitivity towards the rewarding stimulus relative to the placebo group during learning. Computational modeling revealed that losartan reduced the learning rate for negative outcomes and increased exploitatory choice behaviors while preserving learning for positive outcomes. These behavioral patterns were paralleled on the neural level by increased RPE signaling in orbitofrontal-striatal regions and enhanced positive outcome representations in the ventral striatum (VS) following losartan. In the transfer phase, losartan accelerated response times and enhanced VS functional connectivity with left dorsolateral prefrontal cortex when approaching maximum rewards. These findings elucidate the potential of losartan to reduce the impact of negative outcomes during learning and subsequently facilitate motivational approach towards maximum rewards in the transfer of learning. This may indicate a promising therapeutic mechanism to normalize distorted reward learning and fronto-striatal functioning in depression.

NPAS4 in the medial prefrontal cortex mediates chronic social defeat stress-induced anhedonia-like behavior and reductions in excitatory synapses

Chronic stress can produce reward system deficits (i.e., anhedonia) and other common symptoms associated with depressive disorders, as well as neural circuit hypofunction in the medial prefrontal cortex (mPFC). However, the molecular mechanisms by which chronic stress promotes depressive-like behavior and hypofrontality remain unclear. We show here that the neuronal activity-regulated transcription factor, NPAS4, in the mPFC is regulated by chronic social defeat stress (CSDS), and it is required in this brain region for CSDS-induced changes in sucrose preference and natural reward motivation in the mice. Interestingly, NPAS4 is not required for CSDS-induced social avoidance or anxiety-like behavior. We also find that mPFC NPAS4 is required for CSDS-induced reductions in pyramidal neuron dendritic spine density, excitatory synaptic transmission, and presynaptic function, revealing a relationship between perturbation in excitatory synaptic transmission and the expression of anhedonia-like behavior in the mice. Finally, analysis of the mice mPFC tissues revealed that NPAS4 regulates the expression of numerous genes linked to glutamatergic synapses and ribosomal function, the expression of upregulated genes in CSDS-susceptible animals, and differentially expressed genes in postmortem human brains of patients with common neuropsychiatric disorders, including depression. Together, our findings position NPAS4 as a key mediator of chronic stress-induced hypofrontal states and anhedonia-like behavior.

The update of self-identity: Importance of assessing autobiographical memory in major depressive disorder

Major depressive disorder is a leading global cause of disability. There is a growing interest for memory in mood disorders since it might constitute an original tool for prevention, diagnosis, and treatment. MDD is associated with impaired autobiographical memory characterized by a tendency to overgeneral memory, rather than vivid episodic self-defining memory, which is mandatory for problem-solving and projection in the future. This memory bias is maintained by three mechanisms: ruminations, avoidance, and impaired executive control. If we adopt a broader and comprehensive perspective, we can hypothesize that all those alterations have the potential to impair self-identity updating. We posit that this update requires a double referencing process: (1) to internalized self-representation and (2) to an externalized framework dealing with the representation of the consequence of actions. Diagnostic and therapeutic implications are discussed in the light of this model and the importance of assessing autobiographical memory in MDD is highlighted. This article is categorized under: Psychology > Memory Psychology > Brain Function and Dysfunction Neuroscience > Clinical.

The computational psychiatry of antisocial behaviour and psychopathy

Antisocial behaviours such as disobedience, lying, stealing, destruction of property, and aggression towards others are common to multiple disorders of childhood and adulthood, including conduct disorder, oppositional defiant disorder, psychopathy, and antisocial personality disorder. These disorders have a significant negative impact for individuals and for society, but whether they represent clinically different phenomena, or simply different approaches to diagnosing the same underlying psychopathology is highly debated. Computational psychiatry, with its dual focus on identifying different classes of disorder and health (data-driven) and latent cognitive and neurobiological mechanisms (theory-driven), is well placed to address these questions. The elucidation of mechanisms that might characterise latent processes across different disorders of antisocial behaviour can also provide important advances. In this review, we critically discuss the contribution of computational research to our understanding of various antisocial behaviour disorders, and highlight suggestions for how computational psychiatry can address important clinical and scientific questions about these disorders in the future.

Computational models of subjective feelings in psychiatry

Research in computational psychiatry is dominated by models of behavior. Subjective experience during behavioral tasks is not well understood, even though it should be relevant to understanding the symptoms of psychiatric disorders. Here, we bridge this gap and review recent progress in computational models for subjective feelings. For example, happiness reflects not how well people are doing, but whether they are doing better than expected. This dependence on recent reward prediction errors is intact in major depression, although depressive symptoms lower happiness during tasks. Uncertainty predicts subjective feelings of stress in volatile environments. Social prediction errors influence feelings of self-worth more in individuals with low self-esteem despite a reduced willingness to change beliefs due to social feedback. Measuring affective state during behavioral tasks provides a tool for understanding psychiatric symptoms that can be dissociable from behavior. When smartphone tasks are collected longitudinally, subjective feelings provide a potential means to bridge the gap between lab-based behavioral tasks and real-life behavior, emotion, and psychiatric symptoms.

Anhedonia modulates benzodiazepine and opioid demand among persons in treatment for opioid use disorder

Background

Benzodiazepine (BZD) misuse is a significant public health problem, particularly in conjunction with opioid use, due to increased risks of overdose and death. One putative mechanism underlying BZD misuse is affective dysregulation, via exaggerated negative affect (e.g., anxiety, depression, stress-reactivity) and/or impaired positive affect (anhedonia). Similar to other misused substances, BZD consumption is sensitive to price and individual differences. Although purchase tasks and demand curve analysis can shed light on determinants of substance use, few studies have examined BZD demand, nor factors related to demand.

Methods

This ongoing study is examining simulated economic demand for alprazolam (among BZD lifetime misusers based on self-report and DSM-5 diagnosis; n = 23 total; 14 male, 9 female) and each participant's preferred-opioid/route using hypothetical purchase tasks among patients with opioid use disorder (n = 59 total; 38 male, 21 female) who are not clinically stable, i.e., defined as being early in treatment or in treatment longer but with recent substance use. Aims are to determine whether: (1) BZD misusers differ from never-misusers on preferred-opioid economic demand, affective dysregulation (using questionnaire and performance measures), insomnia/behavioral alertness, psychiatric diagnoses or medications, or urinalysis results; and (2) alprazolam demand among BZD misusers is related to affective dysregulation or other measures.

Results

Lifetime BZD misuse is significantly (p < 0.05) related to current major depressive disorder diagnosis, opioid-negative and methadone-negative urinalysis, higher trait anxiety, greater self-reported affective dysregulation, and younger age, but not preferred-opioid demand or insomnia/behavioral alertness. Alprazolam and opioid demand are each significantly positively related to higher anhedonia and, to a lesser extent, depression symptoms but no other measures of negative-affective dysregulation, psychiatric conditions or medications (including opioid agonist therapy or inpatient/outpatient treatment modality), or sleep-related problems.

Conclusion

Anhedonia (positive-affective deficit) robustly predicted increased BZD and opioid demand; these factors could modulate treatment response. Routine assessment and effective treatment of anhedonia in populations with concurrent opioid and sedative use disorder may improve treatment outcomes.

Clinical trial registration

https://clinicaltrials.gov/ct2/show/NCT03696017, identifier NCT03696017.

Sex-dependent effects of early life stress on reinforcement learning and limbic cortico-striatal functional connectivity

Major depressive disorder (MDD) is a stress-related condition hypothesized to involve aberrant reinforcement learning (RL) with positive and negative stimuli. The present study investigated whether repeated early maternal separation (REMS) stress, a procedure widely recognized to cause depression-like behaviour, affects how subjects learn from positive and negative feedback. The REMS procedure was implemented by separating male and female rats from their dam for 6 h each day from post-natal day 5-19. Control rat offspring were left undisturbed during this period. Rats were tested as adults for behavioral flexibility and feedback sensitivity on a probabilistic reversal learning task. A computational approach based on RL theory was used to derive latent behavioral variables related to reward learning and flexibility. To assess underlying brain substrates, a seed-based functional MRI connectivity analysis was applied both before and after an additional adulthood stressor in control and REMS rats. Female but not male rats exposed to REMS stress showed increased response 'stickiness' (repeated responses regardless of reward outcome). Following repeated adulthood stress, reduced functional connectivity from the basolateral amygdala (BLA) to the dorsolateral striatum (DLS), cingulate cortex (Cg), and anterior insula (AI) cortex was observed in females. By contrast, control male rats exposed to the second stressor showed impaired learning from negative feedback (i.e., non-reward) and reduced functional connectivity from the BLA to the DLS and AI compared to maternally separated males. RL in male rats exposed to REMS was unaffected. The fMRI data further revealed that connectivity between the mOFC and other prefrontal cortical and subcortical structures was positively correlated with response 'stickiness'. These findings reveal differences in how females and males respond to early life adversity and subsequent stress. These effects may be mediated by functional divergence in resting-state connectivity between the basolateral amygdala and fronto-striatal brain regions.

Functional MRI study of feedback-based reinforcement learning in depression

Reinforcement learning depends upon the integrity of emotional circuitry to establish associations between environmental cues, decisions, and positive or negative outcomes in order to guide behavior through experience. The emotional dysregulation characteristic of major depressive disorder (MDD) may alter activity in frontal and limbic structures that are key to learning. Although reward and decision-making have been examined in MDD, the effects of depression on associative learning is less well studied. We investigated whether depressive symptoms would be related to abnormalities in learning-related brain activity as measured by functional magnetic resonance imaging (fMRI). Also, we explored whether melancholic and atypical features were associated with altered brain activity. We conducted MRI scans on a 4T Varian MRI system in 10 individuals with MDD and 10 healthy subjects. We examined event-related brain activation during feedback-based learning task using Analysis of Functional NeuroImages (AFNI) for image processing and statistical analysis. We observed that MDD patients exhibited reduced activation in visual cortex but increased activation in cingulate and insular regions compared to healthy participants. Also, in relation to features of depressive subtypes, we observed that levels of activation in striatal, thalamic, and precuneus regions were negatively correlated with atypical characteristics. These results suggest that the effects of MDD change the neural circuitry underlying associative learning, and these effects may depend upon subtype features of MDD.

Decision-making biases in suicide attempters with major depressive disorder: A computational modeling study using the balloon analog risk task (BART)

BACKGROUND

In the last decade, suicidality has been increasingly theorized as a distinct phenomenon from major depressive disorder (MDD), with unique psychological and neural mechanisms, rather than being mostly a severe symptom of MDD. Although decision-making biases have been widely reported in suicide attempters with MDD, little is known regarding what components of these biases can be distinguished from depressiveness itself.

METHODS

Ninety-three patients with current MDD (40 with suicide attempts [SA group] and 53 without suicide attempts [NS group]) and 65 healthy controls (HCs) completed psychometric assessments and the balloon analog risk task (BART). To analyze and compare decision-making components among the three groups, we applied a five-parameter Bayesian computational modeling.

RESULTS

Psychological assessments showed that the SA group had greater suicidal ideation and psychological pain avoidance than the NS group. Computational modeling showed that both MDD groups had higher risk preference and lower ability to learn and adapt from within-task observations than HCs, without differences between the SA and NS patient groups. The SA group also had higher loss aversion than the NS and HC groups, which had similar loss aversion.

CONCLUSIONS

Our BART and computational modeling findings suggest that psychological pain avoidance and loss aversion may be important suicide risk factor that are distinguishable from depression illness itself.

Elevated Effort Cost Identified by Computational Modeling as a Distinctive Feature Explaining Multiple Behaviors in Patients With Depression

BACKGROUND

Motivational deficit is a core clinical manifestation of depression and a strong predictor of treatment failure. However, the underlying mechanisms, which cannot be accessed through conventional questionnaire-based scoring, remain largely unknown. According to decision theory, apathy could result either from biased subjective estimates (of action costs or outcomes) or from dysfunctional processes (in making decisions or allocating resources).

METHODS

Here, we combined a series of behavioral tasks with computational modeling to elucidate the motivational deficits of 35 patients with unipolar or bipolar depression under various treatments compared with 35 matched healthy control subjects.

RESULTS

The most striking feature, which was observed independent of medication across preference tasks (likeability ratings and binary decisions), performance tasks (physical and mental effort exertion), and instrumental learning tasks (updating choices to maximize outcomes), was an elevated sensitivity to effort cost. By contrast, sensitivity to action outcomes (reward and punishment) and task-specific processes were relatively spared.

CONCLUSIONS

These results highlight effort cost as a critical dimension that might explain multiple behavioral changes in patients with depression. More generally, they validate a test battery for computational phenotyping of motivational states, which could orientate toward specific medication or rehabilitation therapy, and thereby help pave the way for more personalized medicine in psychiatry.

Prefrontal cortex activities during verbal fluency and emotional words tasks in major depressive, adjustment, and bipolar disorders with depressive states

BACKGROUND

It can be difficult to differentiate psychiatric disorders from depressive states, with little knowledge on how to differentiate them. This study aimed to evaluate changes in brain activity during cognitive and emotional tasks in patients with depressive state to help with differential diagnoses.

METHODS

Sixty-two patients with depressive states [17 with adjustment disorder (AD), 27 with major depressive disorder (MDD), and 18 with bipolar disorder (BD)] and 34 healthy controls (HC) were recruited. We used a verbal fluency task (VFT) and emotional word tasks with happy and threat words. Functional near-infrared spectroscopy measured the relative change in oxygenated hemoglobin in the frontotemporal areas.

RESULTS

During the VFT, patients with AD or MDD showed significantly reduced activation in the bilateral frontotemporal region (all p < 0.01), whereas patients with BD demonstrated significantly reduced activation in the right frontotemporal areas compared to HC (p < 0.01). During the emotional words task with happy words, patients with MDD showed significantly increased activity in the frontopolar area compared to HC (p = 0.023). Binary logistic regression analysis showed that MDD or BD was significantly associated with brain activity during the happy word task. In distinguishing MDD or BD from HC, the happy words task performed equally well, with an area under the curve of 0.70.

LIMITATIONS

All study patients were taking psychotropic drugs.

CONCLUSIONS

Brain activation in response to a combination of cognitive or emotional stimuli could assist in distinguishing patients with depressive states from healthy controls.

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders

In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.

The neuroanatomy of social trust predicts depression vulnerability

Trust attitude is a social personality trait linked with the estimation of others’ trustworthiness. Trusting others, however, can have substantial negative effects on mental health, such as the development of depression. Despite significant progress in understanding the neurobiology of trust, whether the neuroanatomy of trust is linked with depression vulnerability remains unknown. To investigate a link between the neuroanatomy of trust and depression vulnerability, we assessed trust and depressive symptoms and employed neuroimaging to acquire brain structure data of healthy participants. A high depressive symptom score was used as an indicator of depression vulnerability. The neuroanatomical results observed with the healthy sample were validated in a sample of clinically diagnosed depressive patients. We found significantly higher depressive symptoms among low trusters than among high trusters. Neuroanatomically, low trusters and depressive patients showed similar volume reduction in brain regions implicated in social cognition, including the dorsolateral prefrontal cortex (DLPFC), dorsomedial PFC, posterior cingulate, precuneus, and angular gyrus. Furthermore, the reduced volume of the DLPFC and precuneus mediated the relationship between trust and depressive symptoms. These findings contribute to understanding social- and neural-markers of depression vulnerability and may inform the development of social interventions to prevent pathological depression.

Computational approaches to treatment response prediction in major depression using brain activity and behavioral data: A systematic review

Major depressive disorder is a heterogeneous diagnostic category with multiple available treatments. With the goal of optimizing treatment selection, researchers are developing computational models that attempt to predict treatment response based on various pretreatment measures. In this paper, we review studies that use brain activity data to predict treatment response. Our aim is to highlight and clarify important methodological differences between various studies that relate to the incorporation of domain knowledge, specifically within two approaches delineated as data-driven and theory-driven. We argue that theory-driven generative modeling, which explicitly models information processing in the brain and thus can capture disease mechanisms, is a promising emerging approach that is only beginning to be utilized in treatment response prediction. The predictors extracted via such models could improve interpretability, which is critical for clinical decision-making. We also identify several methodological limitations across the reviewed studies and provide suggestions for addressing them. Namely, we consider problems with dichotomizing treatment outcomes, the importance of investigating more than one treatment in a given study for differential treatment response predictions, the need for a patient-centered approach for defining treatment outcomes, and finally, the use of internal and external validation methods for improving model generalizability.

Computational markers of experience- but not description-based decision-making are associated with future depressive symptoms in young adults

BACKGROUND

Early prediction of high depressive symptoms is crucial for selective intervention and the minimization of functional impairment. Recent cross-sectional studies indicated decision-making deficits in depression, which may be an important contributor to the disorder. Our goal was to test whether description- and experience-based decision making, two major neuroeconomic paradigms of decision-making under uncertainty, predict future depressive symptoms in young adults.

METHODS

One hundred young adults performed two decision-making tasks, one description-based, in which subjects chose between two gambling options given explicitly stated rewards and their probabilities, and the other experience-based, in which subjects were shown rewards but had to learn the probability of those rewards (or cue-outcome contingencies) via trial-and-error experience. We evaluated subjects' depressive symptoms with BDI-II at baseline (T1) and half a year later (T2).

RESULTS

Comparing subjects with low versus high levels of depressive symptoms at T2 showed that the latter performed worse on the experience- but not description-based task at T1. Computational modeling of the decision-making process suggested that subjects with high levels of depressive symptoms had a more concave utility function, indicating enhanced risk aversion. Furthermore, a more concave utility function at T1 increased the odds of high depressive symptoms at T2, even after controlling depressive symptoms at T1, perceived stress at T2, and several covariates (OR = 0.251, 95% CI [0.085, 0.741]).

CONCLUSIONS

This is the first study to demonstrate a prospective link between experience-based decision-making and depressive symptoms. Our results suggest that enhanced risk aversion in experience-based decision-making may be an important contributor to the development of depressive symptoms.

The effect of positive autobiographical memory retrieval on decision-making under risk: A computational model-based analysis

Psychiatric disorders such as depressive and anxiety disorders are associated with altered decision-making under risk. Recent advances in neuroeconomics and computational psychiatry have further discomposed risk-based decision-making into distinct cognitive computational constructs and showed that there may be disorder-specific alterations in these constructs. As a result, it has been suggested these cognitive computational constructs may serve as useful behavioral biomarkers for these disorders. However, to date, little is known about what psychological or behavioral interventions can help to reverse and manage the altered cognitive computational constructs underlying risk-based decision-making. In the present study, we set out to investigate whether recalling positive autobiographical memories may affect risk-based decision-making in healthy volunteers using a description-based task. Specifically, based on theories of behavioral economics, we dissected risk preference into two cognitive computational constructs, utility sensitivity and probability weighting. We found that compared to recalling neutral memories, retrieving positive autobiographical memories increased utility sensitivity (Cohen's d = 0.447), indicating reduced risk aversion. Meanwhile, we also tested the influence of memory retrieval on probability weighting, the effect, however, was unreliable and requires further in-depth investigation. Of clinical relevance, the change in risk aversion after recalling positive memories was in the opposite direction compared to those reported in psychiatric disorders. These results argue for the potential therapeutic effect of positive autobiographical memory retrieval for the amendment of altered risk-based decision-making in psychiatric disorders.

Amygdala-cortical collaboration in reward learning and decision making

Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.

Enhancing the Psychometric Properties of the Iowa Gambling Task Using Full Generative Modeling

Poor psychometrics, particularly low test-retest reliability, pose a major challenge for using behavioral tasks in individual differences research. Here, we demonstrate that full generative modeling of the Iowa Gambling Task (IGT) substantially improves test-retest reliability and may also enhance the IGT's validity for use in characterizing internalizing pathology, compared to the traditional analytic approach. IGT data ( n = 50 ) was collected across two sessions, one month apart. Our full generative model incorporated (1) the Outcome Representation Learning (ORL) computational model at the person-level and (2) a group-level model that explicitly modeled test-retest reliability, along with other group-level effects. Compared to the traditional 'summary score' (proportion good decks selected), the ORL model provides a theoretically rich set of performance metrics (Reward Learning Rate ( A + ) , Punishment Learning Rate ( A - ) , Win Frequency Sensitivity ( β f ) , Perseveration Tendency ( β p ) , Memory Decay ( K ) ), capturing distinct psychological processes. While test-retest reliability for the traditional summary score was only moderate (r = . 37 , BCa 95% CI [.04, .63]), test-retest reliabilities for ORL performance metrics produced by the full generative model were substantially improved, with test-retest correlations ranging between r = . 64 - . 82 for the five ORL parameters. Further, while summary scores showed no substantial associations with internalizing symptoms, ORL parameters were significantly associated with internalizing symptoms. Specifically, Punishment Learning Rate was associated with higher self-reported depression and Perseveration Tendency was associated with lower self-reported anhedonia. Generative modeling offers promise for advancing individual differences research using the IGT, and behavioral tasks more generally, through enhancing task psychometrics.

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

In evaluating our choices, we often suffer from two tragic relativities. First, when our lives change for the better, we rapidly habituate to the higher standard of living. Second, we cannot escape comparing ourselves to various relative standards. Habituation and comparisons can be very disruptive to decision-making and happiness, and till date, it remains a puzzle why they have come to be a part of cognition in the first place. Here, we present computational evidence that suggests that these features might play an important role in promoting adaptive behavior. Using the framework of reinforcement learning, we explore the benefit of employing a reward function that, in addition to the reward provided by the underlying task, also depends on prior expectations and relative comparisons. We find that while agents equipped with this reward function are less happy, they learn faster and significantly outperform standard reward-based agents in a wide range of environments. Specifically, we find that relative comparisons speed up learning by providing an exploration incentive to the agents, and prior expectations serve as a useful aid to comparisons, especially in sparsely-rewarded and non-stationary environments. Our simulations also reveal potential drawbacks of this reward function and show that agents perform sub-optimally when comparisons are left unchecked and when there are too many similar options. Together, our results help explain why we are prone to becoming trapped in a cycle of never-ending wants and desires, and may shed light on psychopathologies such as depression, materialism, and overconsumption.

Even in favorable circumstances, we often find it hard to remain happy with what we have. One might enjoy a newly bought car for a season, but over time it brings fewer positive feelings and one eventually begins dreaming of the next rewarding thing to pursue. Here, we present a series of computational simulations that suggest these presumable “flaws” might play an important role in promoting adaptive behavior. We explore the value of prior expectations and relative comparisons as a useful reward signal and find that across a wide range of environments, these features help an agent learn faster and adapt better to changes in the environment. Our simulations also highlight scenarios when these relative features can be harmful to decision-making and happiness. Together, our results help explain why we have the propensity to keep wanting more, even if it contributes to depression, materialism, and overconsumption.

Reward-based reinforcement learning is altered among individuals with a history of major depressive disorder and psychomotor retardation symptoms

Reward-based reinforcement learning impairments are common in major depressive disorder, but it is unclear which aspects of reward-based reinforcement learning are disrupted in remitted major depression (rMDD). Given that the neurobiological substrates that implement reward-based RL are also strongly implicated in psychomotor retardation (PmR), the present study sought to test whether reward-based reinforcement learning is altered in rMDD individuals with a history of PmR. Three groups of individuals (1) rMDD with past PmR (PmR+, N = 34), (2) rMDD without past PmR (PmR-, N = 44), and (3) healthy controls (N = 90) completed a reward-based reinforcement learning task. Computational modeling was applied to test for group differences in model-derived parameters - specifically, learning rates and reward sensitivity. Compared to controls, rMDD PmR + exhibited lower learning rates, but not reduced reward sensitivity. By contrast, rMDD PmR- did not significantly differ from controls on either of the model-derived parameters. Follow-up analyses indicated that the results were not due to current psychopathology symptoms. Results indicate that a history of PmR predicts altered reward-based reinforcement learning in rMDD. Abnormal reward-related reinforcement learning may reflect a scar of past depressive episodes that contained psychomotor symptoms, or a trait-like deficit that preceded these episodes.

Who are the placebo responders? A cross-sectional cohort study for psychological determinants

ABSTRACT

A number of studies have demonstrated substantial individual differences in placebo effects. We aimed to identify individual psychological factors that potentially predicted the magnitude of placebo hypoalgesia and individual responsiveness. The Research Domain Criteria framework and a classical conditioning with suggestions paradigm were adopted as experimental models to study placebo phenotypes in a cohort of 397 chronic pain participants with a primary diagnosis of temporomandibular disorder (TMD) and 397 healthy control (HC) participants. The magnitude of placebo hypoalgesia was operationalized as the average difference in pain ratings between the placebo and control conditions. The individual placebo responsiveness was identified as the status of placebo responders and nonresponders based on a permutation test. We observed significant placebo effects in both TMD and HC participants. A greater level of emotional distress was a significant predictor of smaller magnitude (slope b = -0.07) and slower extinction rate (slope b = 0.51) of placebo effects in both TMD and HC participants. Greater reward seeking was linked to greater postconditioning expectations (ie, reinforced expectations) in TMD (slope b = 0.16), but there was no such a prediction in HC participants. These findings highlight that negative valence systems might play a role in impairing placebo effects, with a larger impact in chronic pain participants than in healthy participants, suggesting that individuals reporting emotional distress and maladaptive cognitive appraisals of pain may benefit less from placebo effects.

Nonlinear Probability Weighting in Depression and Anxiety: Insights From Healthy Young Adults

Both depressive and anxiety disorders have been associated with excessive risk avoidant behaviors, which are considered an important contributor to the maintenance and recurrence of these disorders. However, given the high comorbidity between the two disorders, their independent association with risk preference remains unclear. Furthermore, due to the involvement of multiple cognitive computational factors in the decision-making tasks employed so far, the precise underlying mechanisms of risk preference are unknown. In the present study, we set out to investigate the common versus unique cognitive computational mechanisms of risk preference in depression and anxiety using a reward-based decision-making task and computational modeling based on economic theories. Specifically, in model-based analysis, we decomposed risk preference into utility sensitivity (a power function) and probability weighting (the one-parameter Prelec weighting function). Multiple linear regression incorporating depression (BDI-II) and anxiety (STAI state anxiety) simultaneously indicated that only depression was associated with one such risk preference parameter, probability weighting. As the symptoms of depression increased, subjects' tendency to overweight small probabilities and underweight large probabilities decreased. Neither depression nor anxiety was associated with utility sensitivity. These associations remained even after controlling covariates or excluding anxiety-relevant items from the depression scale. To our knowledge, this is the first study to assess risk preference due to a concave utility function and nonlinear probability weighting separately for depression and anxiety using computational modeling. Our results provide a mechanistic account of risk avoidance and may improve our understanding of decision-making deficits in depression and anxiety.

Computational Neuroscience Approach to Psychiatry: A Review on Theory-driven Approaches

Translating progress in neuroscience into clinical benefits for patients with psychiatric disorders is challenging because it involves the brain as the most complex organ and its interaction with a complex environment and condition. Dealing with such complexity requires powerful techniques. Computational neuroscience approach to psychiatry integrates multiple levels and types of simulation, analysis and computation according to the different types of computational models to enhance comprehending, prediction and treatment of psychiatric disorder. This approach comprises two approaches: theory-driven and data-driven. In this review, we focus on recent advances in theory-driven approaches that mathematically and mechanistically examine the relationships between disorder-related changes and behavior at different level of brain organization. We discuss recent progresses in computational neuroscience models that relate to psychiatry and show how principles of neural computational modeling can be employed to explain psychopathology.

Beyond Description and Deficits: How Computational Psychiatry Can Enhance an Understanding of Decision-Making in Anorexia Nervosa

PURPOSE OF REVIEW

Despite decades of research, knowledge of the mechanisms maintaining anorexia nervosa (AN) remains incomplete and clearly effective treatments elusive. Novel theoretical frameworks are needed to advance mechanistic and treatment research for this disorder. Here, we argue the utility of engaging a novel lens that differs from existing perspectives in psychiatry. Specifically, we argue the necessity of expanding beyond two historically common perspectives: (1) the descriptive perspective: the tendency to define mechanisms on the basis of surface characteristics and (2) the deficit perspective: the tendency to search for mechanisms associated with under-functioning of decision-making abilities and related circuity, rather than problems of over-functioning, in psychiatric disorders.

RECENT FINDINGS

Computational psychiatry can provide a novel framework for understanding AN because this approach emphasizes the role of computational misalignments (rather than absolute deficits or excesses) between decision-making strategies and environmental demands as the key factors promoting psychiatric illnesses. Informed by this approach, we argue that AN can be understood as a disorder of excess goal pursuit, maintained by over-engagement, rather than disengagement, of executive functioning strategies and circuits. Emerging evidence suggests that this same computational imbalance may constitute an under-investigated phenotype presenting transdiagnostically across psychiatric disorders. A variety of computational models can be used to further elucidate excess goal pursuit in AN. Most traditional psychiatric treatments do not target excess goal pursuit or associated neurocognitive mechanisms. Thus, targeting at the level of computational dysfunction may provide a new avenue for enhancing treatment for AN and related disorders.

Symptom Patterns of the Occurrence of Depression and Anxiety in a Japanese General Adult Population Sample: A Latent Class Analysis

BACKGROUND

Given the high comorbidity and shared risk factors between depression and anxiety, whether they represent theoretically distinct disease entities or are just characteristics of a common negative affect dimension remains debated. Employing a data-driven and person-centered approach, the present study aims to identify meaningful and discrete symptom patterns of the occurrence of depression and anxiety.

METHODS

Using data from an adult sample from the Japanese general population (n = 403, including 184 females, age = 42.28 ± 11.87 years), we applied latent class analysis to identify distinct symptom patterns of depression (PHQ-9) and anxiety (STAI Y1). To empirically validate the derived class memberships, we tested the association between the derived classes and personal profiles including childhood experiences, life events, and personality traits.

RESULTS

The best-fitting solution had four distinct symptom patterns or classes. Whereas both Class 1 and 2 had high depression, Class 1 showed high anxiety due to high anxiety-present symptoms (e.g., "I feel nervous") while Class 2 showed moderate anxiety due to few anxiety-absent symptoms (e.g., "I feel calm"). Class 3 manifested mild anxiety symptoms due to lacking responses on anxiety-absent items. Class 4 manifested the least depressive and anxiety-present symptoms as well as the most anxiety-absent symptoms. Importantly, whereas both Class 1 and 2 had higher childhood neglect and reduced reward responsiveness, etc. compared to Class 4 (i.e., the most healthy class), only Class 1 had greater negative affect and reported more negative life events.

CONCLUSIONS

To our knowledge, this is the first latent class analysis that examined the symptom patterns of depression and anxiety in Asian subjects. The classes we identified have distinct features that confirm their unique patterns of symptom endorsement. Our findings may provide insights into the etiology of depression, anxiety, and their comorbidity.

Investigation of reward learning and feedback sensitivity in non-clinical participants with a history of early life stress

Early life stress (ELS) is an important risk factor for the development of depression. Impairments in reward learning and feedback sensitivity are suggested to be an intermediate phenotype in depression aetiology therefore we hypothesised that healthy adults with a history of ELS would exhibit reward processing deficits independent of any current depressive symptoms. We recruited 64 adults with high levels of ELS and no diagnosis of a current mental health disorder and 65 controls. Participants completed the probabilistic reversal learning task and probabilistic reward task followed by depression, anhedonia, social status, and stress scales. Participants with high levels of ELS showed decreased positive feedback sensitivity in the probabilistic reversal learning task compared to controls. High ELS participants also trended towards possessing a decreased model-free learning rate. This was coupled with a decreased learning ability in the acquisition phase of block 1 following the practice session. Neither group showed a reward induced response bias in the probabilistic reward task however high ELS participants exhibited decreased stimuli discrimination. Overall, these data suggest that healthy participants without a current mental health diagnosis but with high levels of ELS show deficits in positive feedback sensitivity and reward learning in the probabilistic reversal learning task that are distinct from depressed patients. These deficits may be relevant to increased depression vulnerability.