The influence of trial order on learning from reward vs. punishment in a probabilistic categorization task: experimental and computational analyses

Reward and Punishment Learning as Predictors of Cognitive Behavioral Therapy Response in Parkinson's Disease Comorbid with Clinical Depression

Depression is highly comorbid among individuals with Parkinson's Disease (PD), who often experience unique challenges to accessing and benefitting from empirically supported interventions like Cognitive Behavioral Therapy (CBT). Given the role of reward processing in both depression and PD, this study analyzed a subset (N = 25) of participants who participated in a pilot telemedicine intervention of PD-informed CBT, and also completed a Reward- and Punishment-Learning Task (RPLT) at baseline. At the conclusion of CBT, participants were categorized into treatment responders (n = 14) and non-responders (n = 11). Responders learned more optimally from negative rather than positive feedback on the RPLT, while this pattern was reversed in non-responders. Computational modeling suggested group differences in learning rate to negative feedback may drive the observed differences. Overall, the results suggest that a within-subject bias for punishment-based learning might help to predict response to CBT intervention for depression in those with PD.Plain Language Summary Performance on a Computerized Task may predict which Parkinson's Disease Patients benefit from Cognitive Behavioral Treatment of Clinical DepressionWhy was the study done? Clinical depression regularly arises in individuals with Parkinson's Disease (PD) due to the neurobiological changes with the onset and progression of the disease as well as the unique psychosocial difficulties associated with living with a chronic condition. Nonetheless, psychiatric disorders among individuals with PD are often underdiagnosed and likewise undertreated for a variety of reasons. The results of our study have implications about how to improve the accuracy and specificity of mental health treatment recommendations in the future to maximize benefits for individuals with PD, who often face additional barriers to accessing quality mental health treatment.What did the researchers do? We explored whether performance on a computerized task called the Reward- and Punishment-Learning Task (RPLT) helped to predict response to Cognitive Behavioral Therapy (CBT) for depression better than other predictors identified in previous studies. Twenty-five individuals with PD and clinical depression that completed a 10-week telehealth CBT program were assessed for: Demographics (Age, gender, etc.); Clinical information (PD duration, mental health diagnoses, levels of anxiety/depression, etc.); Neurocognitive performance (Memory, processing speed, impulse control, etc.); and RPLT performance.What did the researchers find? A total of 14 participants significantly benefitted from CBT treatment while 11 did not significantly benefit from treatment.There were no differences before treatment in the demographics, clinical information, and neurocognitive performance of those participants who ended up benefitting from the treatment versus those who did not.There were, however, differences before treatment in RPLT performance so that those individuals that benefitted from CBT seemed to learn better from negative feedback.What do the findings mean? Our results suggest that the CBT program benefitted those PD patients with clinical depression that seemed to overall learn best from avoiding punishment rather than obtaining reward which was targeted in CBT by focusing on increasing engagement in rewarding activities. The Reward- and Punishment-Learning Task hence may be a useful tool to help predict treatment response and provide more individualized recommendations on how to best maximize the benefits of psychotherapy for individuals with PD that may struggle to connect to mental health care. Caution is recommended about interpretating these results beyond this study as the overall number of participants was small and the data for this study were collected as part of a previous study so there was no opportunity to include additional measurements of interest.

Altered Reinforcement Learning from Reward and Punishment in Anorexia Nervosa: Evidence from Computational Modeling

OBJECTIVES

Anorexia nervosa (AN) is associated with altered sensitivity to reward and punishment. Few studies have investigated whether this results in aberrant learning. The ability to learn from rewarding and aversive experiences is essential for flexibly adapting to changing environments, yet individuals with AN tend to demonstrate cognitive inflexibility, difficulty set-shifting and altered decision-making. Deficient reinforcement learning may contribute to repeated engagement in maladaptive behavior.

METHODS

This study investigated learning in AN using a probabilistic associative learning task that separated learning of stimuli via reward from learning via punishment. Forty-two individuals with Diagnostic and Statistical Manual of Mental Disorders (DSM)-5 restricting-type AN were compared to 38 healthy controls (HCs). We applied computational models of reinforcement learning to assess group differences in learning, thought to be driven by violations in expectations, or prediction errors (PEs). Linear regression analyses examined whether learning parameters predicted BMI at discharge.

RESULTS

AN had lower learning rates than HC following both positive and negative PE (p < .02), and were less likely to exploit what they had learned. Negative PE on punishment trials predicted lower discharge BMI (p < .001), suggesting individuals with more negative expectancies about avoiding punishment had the poorest outcome.

CONCLUSIONS

This is the first study to show lower rates of learning in AN following both positive and negative outcomes, with worse punishment learning predicting less weight gain. An inability to modify expectations about avoiding punishment might explain persistence of restricted eating despite negative consequences, and suggests that treatments that modify negative expectancy might be effective in reducing food avoidance in AN.

Serotonin modulates asymmetric learning from reward and punishment in healthy human volunteers

Instrumental learning is driven by a history of outcome success and failure. Here, we examined the impact of serotonin on learning from positive and negative outcomes. Healthy human volunteers were assessed twice, once after acute (single-dose), and once after prolonged (week-long) daily administration of the SSRI citalopram or placebo. Using computational modelling, we show that prolonged boosting of serotonin enhances learning from punishment and reduces learning from reward. This valence-dependent learning asymmetry increases subjects' tendency to avoid actions as a function of cumulative failure without leading to detrimental, or advantageous, outcomes. By contrast, no significant modulation of learning was observed following acute SSRI administration. However, differences between the effects of acute and prolonged administration were not significant. Overall, these findings may help explain how serotonergic agents impact on mood disorders.

Perirhinal Cortex is Involved in the Resolution of Learned Approach-Avoidance Conflict Associated with Discrete Objects

The rodent ventral and primate anterior hippocampus have been implicated in approach-avoidance (AA) conflict processing. It is unclear, however, whether this structure contributes to AA conflict detection and/or resolution, and if its involvement extends to conditions of AA conflict devoid of spatial/contextual information. To investigate this, neurologically healthy human participants first learned to approach or avoid single novel visual objects with the goal of maximizing earned points. Approaching led to point gain and loss for positive and negative objects, respectively, whereas avoidance had no impact on score. Pairs of these objects, each possessing nonconflicting (positive-positive/negative-negative) or conflicting (positive-negative) valences, were then presented during functional magnetic resonance imaging. Participants either made an AA decision to score points (Decision task), indicated whether the objects had identical or differing valences (Memory task), or followed a visual instruction to approach or avoid (Action task). Converging multivariate and univariate results revealed that within the medial temporal lobe, perirhinal cortex, rather than the anterior hippocampus, was predominantly associated with object-based AA conflict resolution. We suggest the anterior hippocampus may not contribute equally to all learned AA conflict scenarios and that stimulus information type may be a critical and overlooked determinant of the neural mechanisms underlying AA conflict behavior.

Outcome saliency modulates behavioral decision switching

Goal-directed decision making often requires evaluating the outcomes of our decisions, assessing any gains or losses, learning from performance-related feedback, and deciding whether to alter our future decisions. However, it is unclear how these processes can be influenced by the saliency of an outcome (e.g., when one aspect of the outcome is accentuated more than another). Here we investigated whether decision strategies changed when certain aspects of the task outcome (win/loss or correct/incorrect) became more salient and how our brain encoded such saliency signals. We employed a simple two-alternative forced choice gambling task and quantified the frequency at which participants switched decisions to an alternative option in the subsequent trial after receiving feedback on their current selection. We conducted three experiments. In Experiment 1, we established the baseline decision switching behavior: participants switched more frequently following incorrect trials than correct trials, but there was no significant difference between win and loss trials. In Experiment 2, we highlighted the utility (win or loss) or performance (correct or incorrect) dimension of the chosen outcome and we found that the difference in switching frequency was enlarged along the highlighted dimension. However, Experiment 3 showed that when using non-specific saliency emphasis of the outcome, the saliency effect was abolished. We further conducted simultaneous EEG recordings using specific saliency emphasis and found that the feedback-related negativity, P300, and late positive potential could collectively encode saliency modulation of behavioral switching. Lastly, both the frontal and parietal theta-band power encoded the outcome when it was made more salient. Together, our findings suggest that specific outcome saliency can modulate behavioral decision switching between choices and our results have further revealed the neural signatures underlying such saliency modulation. Altering the saliency of an outcome may change how information is weighed during outcome evaluation and thus influence future decisions.

A possible social relative reward effect: Influences of outcome inequity between rats during operant responding

Social interactions/situations have dramatic influences on motivation. Creating animal models examining these influences promotes a better understanding of the psychological and biological underpinnings of social motivation. Rodents are sensitive to social history/experience during associative conditioning and food-sharing tasks. Would reward-oriented operant behavior be sensitive to social influences by showing a negative contrast-like effect when another organism obtains a greater value outcome? We used a side-by-side arrangement of operant response chambers wherein one animal obtained consistently high reward signaled by a discrete cue. The neighboring, experimental rat experienced different combinations of high and low reward trial sequences. Control conditions included distraction from a conspecific in the neighboring chamber (rat distractor) or cue/food dispenser operating without a conspecific (program distractor) in addition to testing subjects alone. Results support an influence of the other animal actively performing the task on the experimental subject's behavior. Primarily, responding was significantly slower for the low reward trials while the neighboring rat was receiving the higher magnitude reward. The lever-press and not food-cup retrieval latency was significantly slower during exposure to a conspecific neighbor performing the operant task. The effect was not obtained in all session sequences and was more pronounced using longer series of consecutive low reward trials. The slowing effect was also obtained with the program-distractor experience in a different trial sequence. These findings suggest a social-induced negative incentive contrast effect in rats possibly mediated by an outcome inequity process that could have key similarities to complex situational-affective effects on motivation involving frustration or jealously.

Age affects reinforcement learning through dopamine-based learning imbalance and high decision noise-not through Parkinsonian mechanisms

Probabilistic reinforcement learning declines in healthy cognitive aging. While some findings suggest impairments are especially conspicuous in learning from rewards, resembling deficits in Parkinson's disease, others also show impairments in learning from punishments. To reconcile these findings, we tested 252 adults from 3 age groups on a probabilistic reinforcement learning task, analyzed trial-by-trial performance with a Q-reinforcement learning model, and correlated both fitted model parameters and behavior to polymorphisms in dopamine-related genes. Analyses revealed that learning from both positive and negative feedback declines with age but through different mechanisms: when learning from negative feedback, older adults were slower due to noisy decision-making; when learning from positive feedback, they tended to settle for a nonoptimal solution due to an imbalance in learning from positive and negative prediction errors. The imbalance was associated with polymorphisms in the DARPP-32 gene and appeared to arise from mechanisms different from those previously attributed to Parkinson's disease. Moreover, this imbalance predicted previous findings on aging using the Probabilistic Selection Task, which were misattributed to Parkinsonian mechanisms.

Valence and magnitude ambiguity in feedback processing

BACKGROUND

Outcome feedback which indicates behavioral consequences are crucial for reinforcement learning and environmental adaptation. Nevertheless, outcome information in daily life is often totally or partially ambiguous. Studying how people interpret this kind of information would provide important knowledge about the human evaluative system.

METHODS

This study concentrates on the neural processing of partially ambiguous feedback, that is, either its valence or magnitude is unknown to participants. To address this topic, we sequentially presented valence and magnitude information; electroencephalography (EEG) response to each kind of presentation was recorded and analyzed. The event-related potential components feedback-related negativity (FRN) and P3 were used as indices of neural activity.

RESULTS

Consistent with previous literature, the FRN elicited by ambiguous valence was not significantly different from that elicited by negative valence. On the other hand, the FRN elicited by ambiguous magnitude was larger than both the large and small magnitude, indicating the motivation to seek unambiguous magnitude information. The P3 elicited by ambiguous valence and ambiguous magnitude was not significantly different from that elicited by negative valence and small magnitude, respectively, indicating the emotional significance of feedback ambiguity. Finally, the aforementioned effects also manifested in the stage of information integration.

CONCLUSION

These findings indicate both similarities and discrepancies between the processing of valence ambiguity and that of magnitude ambiguity, which may help understand the mechanisms of ambiguous information processing.

Generalized Anxiety Disorder and Social Anxiety Disorder, but Not Panic Anxiety Disorder, Are Associated with Higher Sensitivity to Learning from Negative Feedback: Behavioral and Computational Investigation

Anxiety disorders, including generalized anxiety disorder (GAD), social anxiety disorder (SAD), and panic anxiety disorder (PAD), are a group of common psychiatric conditions. They are characterized by excessive worrying, uneasiness, and fear of future events, such that they affect social and occupational functioning. Anxiety disorders can alter behavior and cognition as well, yet little is known about the particular domains they affect. In this study, we tested the cognitive correlates of medication-free patients with GAD, SAD, and PAD, along with matched healthy participants using a probabilistic category-learning task that allows the dissociation between positive and negative feedback learning. We also fitted all participants' data to a Q-learning model and various actor-critic models that examine learning rate parameters from positive and negative feedback to investigate effects of valence vs. action on performance. SAD and GAD patients were more sensitive to negative feedback than either PAD patients or healthy participants. PAD, SAD, and GAD patients did not differ in positive-feedback learning compared to healthy participants. We found that Q-learning models provide the simplest fit of the data in comparison to other models. However, computational analysis revealed that groups did not differ in terms of learning rate or exploration values. These findings argue that (a) not all anxiety spectrum disorders share similar cognitive correlates, but are rather different in ways that do not link them to the hallmark of anxiety (higher sensitivity to negative feedback); and (b) perception of negative consequences is the core feature of GAD and SAD, but not PAD. Further research is needed to examine the similarities and differences between anxiety spectrum disorders in other cognitive domains and potential implementation of behavioral therapy to remediate cognitive deficits.

The Role of Informative and Ambiguous Feedback in Avoidance Behavior: Empirical and Computational Findings

Avoidance behavior is a critical component of many psychiatric disorders, and as such, it is important to understand how avoidance behavior arises, and whether it can be modified. In this study, we used empirical and computational methods to assess the role of informational feedback and ambiguous outcome in avoidance behavior. We adapted a computer-based probabilistic classification learning task, which includes positive, negative and no-feedback outcomes; the latter outcome is ambiguous as it might signal either a successful outcome (missed punishment) or a failure (missed reward). Prior work with this task suggested that most healthy subjects viewed the no-feedback outcome as strongly positive. Interestingly, in a later version of the classification task, when healthy subjects were allowed to opt out of (i.e. avoid) responding, some subjects (“avoiders”) reliably avoided trials where there was a risk of punishment, but other subjects (“non-avoiders”) never made any avoidance responses at all. One possible interpretation is that the “non-avoiders” valued the no-feedback outcome so positively on punishment-based trials that they had little incentive to avoid. Another possible interpretation is that the outcome of an avoided trial is unspecified and that lack of information is aversive, decreasing subjects’ tendency to avoid. To examine these ideas, we here tested healthy young adults on versions of the task where avoidance responses either did or did not generate informational feedback about the optimal response. Results showed that provision of informational feedback decreased avoidance responses and also decreased categorization performance, without significantly affecting the percentage of subjects classified as “avoiders.” To better understand these results, we used a modified Q-learning model to fit individual subject data. Simulation results suggest that subjects in the feedback condition adjusted their behavior faster following better-than-expected outcomes, compared to subjects in the no-feedback condition. Additionally, in both task conditions, “avoiders” adjusted their behavior faster following worse-than-expected outcomes, and treated the ambiguous no-feedback outcome as less rewarding, compared to non-avoiders. Together, results shed light on the important role of ambiguous and informative feedback in avoidance behavior.

Probabilistic reward- and punishment-based learning in opioid addiction: Experimental and computational data

Addiction is the continuation of a habit in spite of negative consequences. A vast literature gives evidence that this poor decision-making behavior in individuals addicted to drugs also generalizes to laboratory decision making tasks, suggesting that the impairment in decision-making is not limited to decisions about taking drugs. In the current experiment, opioid-addicted individuals and matched controls with no history of illicit drug use were administered a probabilistic classification task that embeds both reward-based and punishment-based learning trials, and a computational model of decision making was applied to understand the mechanisms describing individuals' performance on the task. Although behavioral results showed that opioid-addicted individuals performed as well as controls on both reward- and punishment-based learning, the modeling results suggested subtle differences in how decisions were made between the two groups. Specifically, the opioid-addicted group showed decreased tendency to repeat prior responses, meaning that they were more likely to "chase reward" when expectancies were violated, whereas controls were more likely to stick with a previously-successful response rule, despite occasional expectancy violations. This tendency to chase short-term reward, potentially at the expense of developing rules that maximize reward over the long term, may be a contributing factor to opioid addiction. Further work is indicated to better understand whether this tendency arises as a result of brain changes in the wake of continued opioid use/abuse, or might be a pre-existing factor that may contribute to risk for addiction.