Optimizing vs. matching: response strategy in a probabilistic learning task is associated with negative symptoms of schizophrenia

Impaired Expected Value Computations in Schizophrenia Are Associated With a Reduced Ability to Integrate Reward Probability and Magnitude of Recent Outcomes

BACKGROUND

Motivational deficits in people with schizophrenia (PSZ) are associated with an inability to integrate the magnitude and probability of previous outcomes. The mechanisms that underlie probability-magnitude integration deficits, however, are poorly understood. We hypothesized that increased reliance on "valueless" stimulus-response associations, in lieu of expected value (EV)-based learning, could drive probability-magnitude integration deficits in PSZ with motivational deficits.

METHODS

Healthy volunteers (n = 38) and PSZ (n = 49) completed a learning paradigm consisting of four stimulus pairs. Reward magnitude (3, 2, 1, 0 points) and probability (90%, 80%, 20%, 10%) determined each stimulus's EV. Following a learning phase, new and familiar stimulus pairings were presented. Participants were asked to select stimuli with the highest reward value.

RESULTS

PSZ with high motivational deficits made increasingly less optimal choices as the difference in reward value (probability × magnitude) between two competing stimuli increased. Using a previously validated computational hybrid model, PSZ relied less on EV ("Q-learning") and more on stimulus-response learning ("actor-critic"), which correlated with Scale for the Assessment of Negative Symptoms motivational deficit severity. PSZ specifically failed to represent reward magnitude, consistent with model demonstrations showing that response tendencies in the actor-critic were preferentially driven by reward probability.

CONCLUSIONS

Probability-magnitude deficits in PSZ with motivational deficits arise from underutilization of EV in favor of reliance on valueless stimulus-response associations. Confirmed by our computational hybrid framework, probability-magnitude integration deficits were driven specifically by a failure to represent reward magnitude. This work provides a first mechanistic explanation of complex EV-based learning deficits in PSZ with motivational deficits that arise from an inability to combine information from different reward modalities.

The NEWMEDS rodent touchscreen test battery for cognition relevant to schizophrenia

RATIONALE

The NEWMEDS initiative (Novel Methods leading to New Medications in Depression and Schizophrenia, http://www.newmeds-europe.com ) is a large industrial-academic collaborative project aimed at developing new methods for drug discovery for schizophrenia. As part of this project, Work package 2 (WP02) has developed and validated a comprehensive battery of novel touchscreen tasks for rats and mice for assessing cognitive domains relevant to schizophrenia.

OBJECTIVES

This article provides a review of the touchscreen battery of tasks for rats and mice for assessing cognitive domains relevant to schizophrenia and highlights validation data presented in several primary articles in this issue and elsewhere.

METHODS

The battery consists of the five-choice serial reaction time task and a novel rodent continuous performance task for measuring attention, a three-stimulus visual reversal and the serial visual reversal task for measuring cognitive flexibility, novel non-matching to sample-based tasks for measuring spatial working memory and paired-associates learning for measuring long-term memory.

RESULTS

The rodent (i.e. both rats and mice) touchscreen operant chamber and battery has high translational value across species due to its emphasis on construct as well as face validity. In addition, it offers cognitive profiling of models of diseases with cognitive symptoms (not limited to schizophrenia) through a battery approach, whereby multiple cognitive constructs can be measured using the same apparatus, enabling comparisons of performance across tasks.

CONCLUSION

This battery of tests constitutes an extensive tool package for both model characterisation and pre-clinical drug discovery.

Inferring reward prediction errors in patients with schizophrenia: a dynamic reward task for reinforcement learning

Abnormalities in the dopamine system have long been implicated in explanations of reinforcement learning and psychosis. The updated reward prediction error (RPE)—a discrepancy between the predicted and actual rewards—is thought to be encoded by dopaminergic neurons. Dysregulation of dopamine systems could alter the appraisal of stimuli and eventually lead to schizophrenia. Accordingly, the measurement of RPE provides a potential behavioral index for the evaluation of brain dopamine activity and psychotic symptoms. Here, we assess two features potentially crucial to the RPE process, namely belief formation and belief perseveration, via a probability learning task and reinforcement-learning modeling. Forty-five patients with schizophrenia [26 high-psychosis and 19 low-psychosis, based on their p1 and p3 scores in the positive-symptom subscales of the Positive and Negative Syndrome Scale (PANSS)] and 24 controls were tested in a feedback-based dynamic reward task for their RPE-related decision making. While task scores across the three groups were similar, matching law analysis revealed that the reward sensitivities of both psychosis groups were lower than that of controls. Trial-by-trial data were further fit with a reinforcement learning model using the Bayesian estimation approach. Model fitting results indicated that both psychosis groups tend to update their reward values more rapidly than controls. Moreover, among the three groups, high-psychosis patients had the lowest degree of choice perseveration. Lumping patients' data together, we also found that patients' perseveration appears to be negatively correlated (p = 0.09, trending toward significance) with their PANSS p1 + p3 scores. Our method provides an alternative for investigating reward-related learning and decision making in basic and clinical settings.

Orbitofrontal dopamine depletion upregulates caudate dopamine and alters behavior via changes in reinforcement sensitivity

Schizophrenia is associated with upregulation of dopamine (DA) release in the caudate nucleus. The caudate has dense connections with the orbitofrontal cortex (OFC) via the frontostriatal loops, and both areas exhibit pathophysiological change in schizophrenia. Despite evidence that abnormalities in dopaminergic neurotransmission and prefrontal cortex function co-occur in schizophrenia, the influence of OFC DA on caudate DA and reinforcement processing is poorly understood. To test the hypothesis that OFC dopaminergic dysfunction disrupts caudate dopamine function, we selectively depleted dopamine from the OFC of marmoset monkeys and measured striatal extracellular dopamine levels (using microdialysis) and dopamine D2/D3 receptor binding (using positron emission tomography), while modeling reinforcement-related behavior in a discrimination learning paradigm. OFC dopamine depletion caused an increase in tonic dopamine levels in the caudate nucleus and a corresponding reduction in D2/D3 receptor binding. Computational modeling of behavior showed that the lesion increased response exploration, reducing the tendency to persist with a recently chosen response side. This effect is akin to increased response switching previously seen in schizophrenia and was correlated with striatal but not OFC D2/D3 receptor binding. These results demonstrate that OFC dopamine depletion is sufficient to induce striatal hyperdopaminergia and changes in reinforcement learning relevant to schizophrenia.

A review of reward processing and motivational impairment in schizophrenia

This article reviews and synthesizes research on reward processing in schizophrenia, which has begun to provide important insights into the cognitive and neural mechanisms associated with motivational impairments. Aberrant cortical-striatal interactions may be involved with multiple reward processing abnormalities, including: (1) dopamine-mediated basal ganglia systems that support reinforcement learning and the ability to predict cues that lead to rewarding outcomes; (2) orbitofrontal cortex-driven deficits in generating, updating, and maintaining value representations; (3) aberrant effort-value computations, which may be mediated by disrupted anterior cingulate cortex and midbrain dopamine functioning; and (4) altered activation of the prefrontal cortex, which is important for generating exploratory behaviors in environments where reward outcomes are uncertain. It will be important for psychosocial interventions targeting negative symptoms to account for abnormalities in each of these reward processes, which may also have important interactions; suggestions for novel behavioral intervention strategies that make use of external cues, reinforcers, and mobile technology are discussed.

Reinforcement learning and dopamine in schizophrenia: dimensions of symptoms or specific features of a disease group?

Abnormalities in reinforcement learning are a key finding in schizophrenia and have been proposed to be linked to elevated levels of dopamine neurotransmission. Behavioral deficits in reinforcement learning and their neural correlates may contribute to the formation of clinical characteristics of schizophrenia. The ability to form predictions about future outcomes is fundamental for environmental interactions and depends on neuronal teaching signals, like reward prediction errors. While aberrant prediction errors, that encode non-salient events as surprising, have been proposed to contribute to the formation of positive symptoms, a failure to build neural representations of decision values may result in negative symptoms. Here, we review behavioral and neuroimaging research in schizophrenia and focus on studies that implemented reinforcement learning models. In addition, we discuss studies that combined reinforcement learning with measures of dopamine. Thereby, we suggest how reinforcement learning abnormalities in schizophrenia may contribute to the formation of psychotic symptoms and may interact with cognitive deficits. These ideas point toward an interplay of more rigid versus flexible control over reinforcement learning. Pronounced deficits in the flexible or model-based domain may allow for a detailed characterization of well-established cognitive deficits in schizophrenia patients based on computational models of learning. Finally, we propose a framework based on the potentially crucial contribution of dopamine to dysfunctional reinforcement learning on the level of neural networks. Future research may strongly benefit from computational modeling but also requires further methodological improvement for clinical group studies. These research tools may help to improve our understanding of disease-specific mechanisms and may help to identify clinically relevant subgroups of the heterogeneous entity schizophrenia.

Measuring reinforcement learning and motivation constructs in experimental animals: relevance to the negative symptoms of schizophrenia

The present review article summarizes and expands upon the discussions that were initiated during a meeting of the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS; http://cntrics.ucdavis.edu) meeting. A major goal of the CNTRICS meeting was to identify experimental procedures and measures that can be used in laboratory animals to assess psychological constructs that are related to the psychopathology of schizophrenia. The issues discussed in this review reflect the deliberations of the Motivation Working Group of the CNTRICS meeting, which included most of the authors of this article as well as additional participants. After receiving task nominations from the general research community, this working group was asked to identify experimental procedures in laboratory animals that can assess aspects of reinforcement learning and motivation that may be relevant for research on the negative symptoms of schizophrenia, as well as other disorders characterized by deficits in reinforcement learning and motivation. The tasks described here that assess reinforcement learning are the Autoshaping Task, Probabilistic Reward Learning Tasks, and the Response Bias Probabilistic Reward Task. The tasks described here that assess motivation are Outcome Devaluation and Contingency Degradation Tasks and Effort-Based Tasks. In addition to describing such methods and procedures, the present article provides a working vocabulary for research and theory in this field, as well as an industry perspective about how such tasks may be used in drug discovery. It is hoped that this review can aid investigators who are conducting research in this complex area, promote translational studies by highlighting shared research goals and fostering a common vocabulary across basic and clinical fields, and facilitate the development of medications for the treatment of symptoms mediated by reinforcement learning and motivational deficits.

A Match Made by Modafinil: Probability Matching in Choice Decisions and Spatial Attention

When predicting where a target or reward will be, participants tend to choose each location commensurate with the true underlying probability (i.e., probability match). The strategy of probability matching involves independent sampling of high and low probability locations on separate trials. In contrast, models of probabilistic spatial attention hypothesize that on any given trial attention will either be weighted toward the high probability location or be distributed equally across all locations. Thus, the strategies of probabilistic sampling by choice decisions and spatial attention appear to differ with regard to low-probability events. This distinction is somewhat surprising because similar brain mechanisms (e.g., pFC-mediated cognitive control) are thought to be important in both functions. Thus, the goal of the current study was to examine the relationship between choice decisions and attentional selection within single trials to test for any strategic differences, then to determine whether that relationship is malleable to manipulations of catecholamine-modulated cognitive control with the drug modafinil. Our results demonstrate that spatial attention and choice decisions followed different strategies of probabilistic information selection on placebo, but that modafinil brought the pattern of spatial attention into alignment with that of predictive choices. Modafinil also produced earlier learning of the probability distribution. Together, these results suggest that enhancing cognitive control mechanisms (e.g., through prefrontal cortical function) leads spatial attention to follow choice decisions in selecting information according to rule-based expectations.

From movement to thought: the development of executive function

This article presents a very simple definition of executive functioning (EF). Although EF is traditionally understood as a cognitive function dependent upon top-down cortical control, we challenge this model. We propose that the functional architecture of the brain evolved to meet the needs of interactive behavior and that cognition develops to control the motor system, which is of paramount importance in adaptation, essentially a manifestation of EF. We propose that traditional models of cognition are incomplete characterizations of EF and that procedural learning and "automatic" behaviors are the most basic, bottom-up functions that support all EF. We propose that motor development in children demonstrates how all knowledge is grounded in sensorimotor interaction and how interactive behavior generates both procedural and declarative knowledge, which later interact to generate EF. This model emphasizes the critical importance of motor behavior in children and stresses the importance of the pediatric motor examination in understanding the development of EF. This model also has implications for why traditional tests of EF have little predictive validity in both children and adults.

Deficits in positive reinforcement learning and uncertainty-driven exploration are associated with distinct aspects of negative symptoms in schizophrenia

BACKGROUND

Negative symptoms are core features of schizophrenia (SZ); however, the cognitive and neural basis for individual negative symptom domains remains unclear. Converging evidence suggests a role for striatal and prefrontal dopamine in reward learning and the exploration of actions that might produce outcomes that are better than the status quo. The current study examines whether deficits in reinforcement learning and uncertainty-driven exploration predict specific negative symptom domains.

METHODS

We administered a temporal decision-making task, which required trial-by-trial adjustment of reaction time to maximize reward receipt, to 51 patients with SZ and 39 age-matched healthy control subjects. Task conditions were designed such that expected value (probability × magnitude) increased, decreased, or remained constant with increasing response times. Computational analyses were applied to estimate the degree to which trial-by-trial responses are influenced by reinforcement history.

RESULTS

Individuals with SZ showed impaired Go learning but intact NoGo learning relative to control subjects. These effects were most pronounced in patients with higher levels of negative symptoms. Uncertainty-based exploration was substantially reduced in individuals with SZ and selectively correlated with clinical ratings of anhedonia.

CONCLUSIONS

Schizophrenia patients, particularly those with high negative symptoms, failed to speed reaction times to increase positive outcomes and showed reduced tendency to explore when alternative actions could lead to better outcomes than the status quo. Results are interpreted in the context of current computational, genetic, and pharmacological data supporting the roles of striatal and prefrontal dopamine in these processes.