Reinforcement learning as an intermediate phenotype in psychosis? Deficits sensitive to illness stage but not associated with polygenic risk of schizophrenia in the general population

Pavlovian impatience: The anticipation of immediate rewards increases approach behaviour

People often exhibit intertemporal impatience by choosing immediate small over delayed larger rewards, which has been implicated across maladaptive behaviours and mental health symptoms. In this preregistered study, we tested the role of an intertemporal Pavlovian bias as possible psychological mechanism driving the temptation posed by immediate rewards. Concretely, we hypothesized that the anticipation of immediate rewards (compared with preference-matched delayed rewards) enhances goal-directed approach behaviour but interferes with goal-directed inhibition. Such a mechanism could contribute to the difficulty to inhibit ourselves in the face of immediate rewards (e.g., a drug), at the cost of long-term (e.g., health) goals. A sample of 184 participants completed a newly developed reinforcement learning go/no-go task with four trial types: Go to win immediate reward; Go to win delayed reward; No-go to win immediate reward; and No-go to win delayed reward trials. Go responding was increased in trials in which an immediate reward was available compared with trials in which a preference-matched delayed reward was available. Computational models showed that on average, this behavioural pattern was best captured by a cue-response bias reflecting a stronger elicitation of go responses upon presentation of an immediate (versus delayed) reward cue. The results of this study support the role of an intertemporal Pavlovian bias as a psychological mechanism contributing to impatient intertemporal choice.

Cognitive subgroups of affective and non-affective psychosis show differences in medication and cortico-subcortical brain networks

Cognitive deficits are prevalent in individuals with psychosis and are associated with neurobiological changes, potentially serving as an endophenotype for psychosis. Using the HCP-Early-Psychosis-dataset (n = 226), we aimed to investigate cognitive subtypes (deficit/intermediate/spared) through data-driven clustering in affective (AP) and non-affective psychosis patients (NAP) and controls (HC). We explored differences between three clusters in symptoms, cognition, medication, and grey matter volume. Applying principal component analysis, we selected features for clustering. Features that explained most variance were scores for intelligence, verbal recognition and comprehension, auditory attention, working memory, reasoning and executive functioning. Fuzzy K-Means clustering on those features revealed that the subgroups significantly varied in cognitive impairment, clinical symptoms, and, importantly, also in medication and grey matter volume in fronto-parietal and subcortical networks. The spared cluster (86%HC, 37%AP, 17%NAP) exhibited unimpaired cognition, lowest symptoms/medication, and grey matter comparable to controls. The deficit cluster (4%HC, 10%AP, 47%NAP) had impairments across all domains, highest symptoms scores/medication dosage, and pronounced grey matter alterations. The intermediate deficit cluster (11%HC, 54%AP, 36%NAP) showed fewer deficits than the second cluster, but similar symptoms/medication/grey matter to the spared cluster. Controlling for medication, cognitive scores correlated with grey matter changes and negative symptoms across all patients. Our findings generally emphasize the interplay between cognition, brain structure, symptoms, and medication in AP and NAP, and specifically suggest a possible mediating role of cognition, highlighting the potential of screening cognitive changes to aid tailoring treatments and interventions.

Distinct alterations in probabilistic reversal learning across at-risk mental state, first episode psychosis and persistent schizophrenia

We used a probabilistic reversal learning task to examine prediction error-driven belief updating in three clinical groups with psychosis or psychosis-like symptoms. Study 1 compared people with at-risk mental state and first episode psychosis (FEP) to matched controls. Study 2 compared people diagnosed with treatment-resistant schizophrenia (TRS) to matched controls. The design replicated our previous work showing ketamine-related perturbations in how meta-level confidence maintained behavioural policy. We applied the same computational modelling analysis here, in order to compare the pharmacological model to three groups at different stages of psychosis. Accuracy was reduced in FEP, reflecting increased tendencies to shift strategy following probabilistic errors. The TRS group also showed a greater tendency to shift choice strategies though accuracy levels were not significantly reduced. Applying the previously-used computational modelling approach, we observed that only the TRS group showed altered confidence-based modulation of responding, previously observed under ketamine administration. Overall, our behavioural findings demonstrated resemblance between clinical groups (FEP and TRS) and ketamine in terms of a reduction in stabilisation of responding in a noisy environment. The computational analysis suggested that TRS, but not FEP, replicates ketamine effects but we consider the computational findings preliminary given limitations in performance of the model.

Associative learning and facial expression recognition in schizophrenic patients: Effects of social presence

Schizophrenia (SCZ) is a psychiatric disorder that alters both general and social cognition. However, the exact mechanisms that are altered remain to be elucidated. In this study, we investigated associative learning (AL) and facial expression recognition (FER) in the same patients, using emotional expressions and abstract images. Our main aim was to investigate how these cognitive abilities are affected by SCZ and to assess the role of mere social presence, a factor that has not been considered before. The study compared the behavioral performance of 60 treated outpatients with SCZ and 103 demographically matched healthy volunteers. In the AL task, participants had to associate abstract images or facial expressions with key presses, guided by feedback on each trial. In the FER task, they had to report whether two successively presented facial expressions were the same or different. All participants performed the two tasks under two social context conditions: alone or in the presence of a passive but attentive audience. The results showed a severe learning impairment in patients compared to controls, with a slight advantage for facial expressions compared to abstract images, and a gender-dependent effect of social presence. In contrast, facial expression recognition was partially spared in patients and facilitated by social presence. We conclude that cognitive abilities are impaired in patients with SCZ, but their investigation needs to take into account the social context in which they are assessed.

Reversal learning in those with early psychosis features contingency-dependent changes in loss response and learning

INTRODUCTION

People with psychotic disorders commonly feature broad decision-making impairments that impact their functional outcomes. Specific associative/reinforcement learning problems have been demonstrated in persistent psychosis. But these phenotypes may differ in early psychosis, suggesting that aspects of cognition decline over time.

METHODS

The present proof-of-concept study examined goal-directed action and reversal learning in controls and those with early psychosis.

RESULTS

Equivalent performance was observed between groups during outcome-specific devaluation, and reversal learning at an 80:20 contingency (reward probability for high:low targets). But when the low target reward probability was increased (80:40) those with early psychosis altered their response to loss, whereas controls did not. Computational modelling confirmed that in early psychosis there was a change in punishment learning that increased the chance of staying with the same stimulus after a loss, multiple trials into the future. In early psychosis, the magnitude of this response was greatest in those with higher IQ and lower clinical severity scores.

CONCLUSIONS

We show preliminary evidence that those with early psychosis present with a phenotype that includes altered responding to loss and hyper-adaptability in response to outcome changes. This may reflect a compensatory response to overcome the milieu of corticostriatal changes associated with psychotic disorders.

Association between increased anterior cingulate glutamate and psychotic-like experiences, but not autistic traits in healthy volunteers

Despite many differences, autism spectrum disorder and schizophrenia spectrum disorder share environmental risk factors, genetic predispositions as well as neuronal abnormalities, and show similar cognitive deficits in working memory, perspective taking, or response inhibition. These shared abnormalities are already present in subclinical traits of these disorders. The literature proposes that changes in the inhibitory GABAergic and the excitatory glutamatergic system could explain underlying neuronal commonalities and differences. Using magnetic resonance spectroscopy (1H-MRS), we investigated the associations between glutamate concentrations in the anterior cingulate cortex (ACC), the left/right putamen, and left/right dorsolateral prefrontal cortex and psychotic-like experiences (Schizotypal Personality Questionnaire) and autistic traits (Autism Spectrum Quotient) in 53 healthy individuals (26 women). To investigate the contributions of glutamate concentrations in different cortical regions to symptom expression and their interactions, we used linear regression analyses. We found that only glutamate concentration in the ACC predicted psychotic-like experiences, but not autistic traits. Supporting this finding, a binomial logistic regression predicting median-split high and low risk groups for psychotic-like experiences revealed ACC glutamate levels as a significant predictor for group membership. Taken together, this study provides evidence that glutamate levels in the ACC are specifically linked to the expression of psychotic-like experiences, and may be a potential candidate in identifying early risk individuals prone to developing psychotic-like experiences.

The role of pharmacogenomics in precision psychiatry

The field of psychiatry is facing an important paradigm shift in the provision of clinical care and mental health service organization toward personalization and integration of multimodal data science. This approach, termed precision psychiatry, aims at identifying subgroups of patients more prone to the development of a certain phenotype, such as symptoms or severe mental disorders (risk detection), and/or to guide treatment selection. Pharmacogenomics and computational psychiatry are two fundamental tools of precision psychiatry, which have seen increasing levels of integration in clinical settings. Here we present a brief overview of these two applications of precision psychiatry in clinical settings.

Belief updating in psychosis, depression and anxiety disorders: A systematic review across computational modelling approaches

Alterations in belief updating are proposed to underpin symptoms of psychiatric illness, including psychosis, depression, and anxiety. Key parameters underlying belief updating can be captured using computational modelling techniques, aiding the identification of unique and shared deficits, and improving diagnosis and treatment. We systematically reviewed research that applied computational modelling to probabilistic tasks measuring belief updating in stable and volatile (changing) environments, across clinical and subclinical psychosis (n = 17), anxiety (n = 9), depression (n = 9) and transdiagnostic samples (n = 9). Depression disorders related to abnormal belief updating in response to the valence of rewards, evidenced in both stable and volatile environments. Whereas psychosis and anxiety disorders were associated with difficulties adapting to changing contingencies specifically, indicating an inflexibility and/or insensitivity to environmental volatility. Higher-order learning models revealed additional difficulties in the estimation of overall environmental volatility across psychosis disorders, showing increased updating to irrelevant information. These findings stress the importance of investigating belief updating in transdiagnostic samples, using homogeneous experimental and computational modelling approaches.

Negative symptoms, striatal dopamine and model-free reward decision-making in schizophrenia

Negative symptoms, such as lack of motivation or social withdrawal, are highly prevalent and debilitating in patients with schizophrenia. Underlying mechanisms of negative symptoms are incompletely understood, thereby preventing the development of targeted treatments. We hypothesized that in patients with schizophrenia during psychotic remission, impaired influences of both model-based and model-free reward predictions on decision-making ('reward prediction influence', RPI) underlie negative symptoms. We focused on psychotic remission, because psychotic symptoms might confound reward-based decision-making. Moreover, we hypothesized that impaired model-based/model-free RPIs depend on alterations of both associative striatum dopamine synthesis and storage (DSS) and executive functioning. Both factors influence RPI in healthy subjects and are typically impaired in schizophrenia. Twenty-five patients with schizophrenia with pronounced negative symptoms during psychotic remission and 24 healthy controls were included in the study. Negative symptom severity was measured by the Positive and Negative Syndrome Scale negative subscale, model-based/model-free RPI by the two-stage decision task, associative striatum DSS by 18F-DOPA positron emission tomography and executive functioning by the symbol coding task. Model-free RPI was selectively reduced in patients and associated with negative symptom severity as well as with reduced associative striatum DSS (in patients only) and executive functions (both in patients and controls). In contrast, model-based RPI was not altered in patients. Results provide evidence for impaired model-free reward prediction influence as a mechanism for negative symptoms in schizophrenia as well as for reduced associative striatum dopamine and executive dysfunction as relevant factors. Data suggest potential treatment targets for patients with schizophrenia and pronounced negative symptoms.

Action selection in early stages of psychosis: an active inference approach

BACKGROUND

To interact successfully with their environment, humans need to build a model to make sense of noisy and ambiguous inputs. An inaccurate model, as suggested to be the case for people with psychosis, disturbs optimal action selection. Recent computational models, such as active inference, have emphasized the importance of action selection, treating it as a key part of the inferential process. Based on an active inference framework, we sought to evaluate previous knowledge and belief precision in an action-based task, given that alterations in these parameters have been linked to the development of psychotic symptoms. We further sought to determine whether task performance and modelling parameters would be suitable for classification of patients and controls.

METHODS

Twenty-three individuals with an at-risk mental state, 26 patients with first-episode psychosis and 31 controls completed a probabilistic task in which action choice (go/no-go) was dissociated from outcome valence (gain or loss). We evaluated group differences in performance and active inference model parameters and performed receiver operating characteristic (ROC) analyses to assess group classification.

RESULTS

We found reduced overall performance in patients with psychosis. Active inference modelling revealed that patients showed increased forgetting, reduced confidence in policy selection and less optimal general choice behaviour, with poorer action-state associations. Importantly, ROC analysis showed fair-to-good classification performance for all groups, when combining modelling parameters and performance measures.

LIMITATIONS

The sample size is moderate.

CONCLUSION

Active inference modelling of this task provides further explanation for dysfunctional mechanisms underlying decision-making in psychosis and may be relevant for future research on the development of biomarkers for early identification of psychosis.

The prediction-error hypothesis of schizophrenia: new data point to circuit-specific changes in dopamine activity

Schizophrenia is a severe psychiatric disorder affecting 21 million people worldwide. People with schizophrenia suffer from symptoms including psychosis and delusions, apathy, anhedonia, and cognitive deficits. Strikingly, schizophrenia is characterised by a learning paradox involving difficulties learning from rewarding events, whilst simultaneously 'overlearning' about irrelevant or neutral information. While dysfunction in dopaminergic signalling has long been linked to the pathophysiology of schizophrenia, a cohesive framework that accounts for this learning paradox remains elusive. Recently, there has been an explosion of new research investigating how dopamine contributes to reinforcement learning, which illustrates that midbrain dopamine contributes in complex ways to reinforcement learning, not previously envisioned. This new data brings new possibilities for how dopamine signalling contributes to the symptomatology of schizophrenia. Building on recent work, we present a new neural framework for how we might envision specific dopamine circuits contributing to this learning paradox in schizophrenia in the context of models of reinforcement learning. Further, we discuss avenues of preclinical research with the use of cutting-edge neuroscience techniques where aspects of this model may be tested. Ultimately, it is hoped that this review will spur to action more research utilising specific reinforcement learning paradigms in preclinical models of schizophrenia, to reconcile seemingly disparate symptomatology and develop more efficient therapeutics.

The Role of Dopaminergic Genes in Probabilistic Reinforcement Learning in Schizophrenia Spectrum Disorders

Schizophrenia spectrum disorders (SZ) are characterized by impairments in probabilistic reinforcement learning (RL), which is associated with dopaminergic circuitry encompassing the prefrontal cortex and basal ganglia. However, there are no studies examining dopaminergic genes with respect to probabilistic RL in SZ. Thus, the aim of our study was to examine the impact of dopaminergic genes on performance assessed by the Probabilistic Selection Task (PST) in patients with SZ in comparison to healthy control (HC) subjects. In our study, we included 138 SZ patients and 188 HC participants. Genetic analysis was performed with respect to the following genetic polymorphisms: rs4680 in COMT, rs907094 in DARP-32, rs2734839, rs936461, rs1800497, and rs6277 in DRD2, rs747302 and rs1800955 in DRD4 and rs28363170 and rs2975226 in DAT1 genes. The probabilistic RL task was completed by 59 SZ patients and 95 HC subjects. SZ patients performed significantly worse in acquiring reinforcement contingencies during the task in comparison to HCs. We found no significant association between genetic polymorphisms and RL among SZ patients; however, among HC participants with respect to the DAT1 rs28363170 polymorphism, individuals with 10-allele repeat genotypes performed better in comparison to 9-allele repeat carriers. The present study indicates the relevance of the DAT1 rs28363170 polymorphism in RL in HC participants.

Neural Circuitry of Salience and Reward Processing in Psychosis

The processing of salient and rewarding stimuli is integral to engaging our attention, stimulating anticipation for future events, and driving goal-directed behaviors. Widespread impairments in these processes are observed in psychosis, which may be associated with worse functional outcomes or mechanistically linked to the development of symptoms. Here, we summarize the current knowledge of behavioral and functional neuroimaging in salience, prediction error, and reward. Although each is a specific process, they are situated in multiple feedback and feedforward systems integral to decision making and cognition more generally. We argue that the origin of salience and reward processing dysfunctions may be centered in the subcortex during the earliest stages of psychosis, with cortical abnormalities being initially more spared but becoming more prominent in established psychotic illness/schizophrenia. The neural circuits underpinning salience and reward processing may provide targets for delaying or preventing progressive behavioral and neurobiological decline.

An informatics approach to examine decision-making impairments in the daily life of individuals with depression

Mental health informatics studies methods that collect, model, and interpret a wide variety of data to generate useful information with theoretical or clinical relevance to improve mental health and mental health care. This article presents a mental health informatics approach that is based on the decision-making theory of depression, whereby daily life data from a natural sequential decision-making task are collected and modeled using a reinforcement learning method. The model parameters are then estimated to uncover specific aspects of decision-making impairment in individuals with depression. Empirical results from a pilot study conducted to examine decision-making impairments in the daily lives of university students with depression are presented to illustrate this approach. Future research can apply and expand on this approach to investigate a variety of daily life situations and psychiatric conditions and to facilitate new informatics applications. Using this approach in mental health research may generate useful information with both theoretical and clinical relevance and high ecological validity.

The impact of social anxiety on feedback-based go and nogo learning

The term “Pavlovian” bias describes the phenomenon that learning to execute a response to obtain a reward or to inhibit a response to avoid punishment is much easier than learning the reverse. The present study investigated the interplay between this learning bias and individual levels of social anxiety. Since avoidance behavior is a hallmark feature of social anxiety and high levels of social anxiety have been associated with better learning from negative feedback, it is conceivable that the Pavlovian bias is altered in individuals with high social anxiety, with a strong tendency to avoid negative feedback, especially (but not only) in a nogo context. In addition, learning may be modulated by the individual propensity to learn from positive or negative feedback, which can be assessed as a trait-like feature. A sample of 84 healthy university students completed an orthogonalized go/nogo task that decoupled action type (go/nogo) and outcome valence (win/avoid) and a probabilistic selection task based upon which the individual propensity to learn from positive and negative feedback was determined. Self-reported social anxiety and learning propensity were used as predictors in linear mixed-effect model analysis of performance accuracy in the go/nogo task. Results revealed that high socially anxious subjects with a propensity to learn better from negative feedback showed particularly pronounced learning for nogo to avoid while lacking significant learning for nogo to win as well as go to avoid. This result pattern suggests that high levels of social anxiety in concert with negative learning propensity hamper the overcoming of Pavlovian bias in a win context while facilitating response inhibition in an avoidance context. The present data confirm the robust Pavlovian bias in feedback-based learning and add to a growing body of evidence for modulation of feedback learning by individual factors, such as personality traits. Specifically, results show that social anxiety is associated with altered Pavlovian bias, and might suggest that this effect could be driven by altered basal ganglia function primarily affecting the nogo pathway.