Probabilistic Reinforcement Learning and Anhedonia

The computational structure of consummatory anhedonia

Anhedonia is a reduction in enjoyment, motivation, or interest. It is common across mental health disorders and a harbinger of poor treatment outcomes. The enjoyment aspect, termed 'consummatory anhedonia', in particular poses fundamental questions about how the brain constructs rewards: what processes determine how intensely a reward is experienced? Here, we outline limitations of existing computational conceptualisations of consummatory anhedonia. We then suggest a richer reinforcement learning (RL) account of consummatory anhedonia with a reconceptualisation of subjective hedonic experience in terms of goal progress. This accounts qualitatively for the impact of stress, dysfunctional cognitions, and maladaptive beliefs on hedonic experience. The model also offers new views on the treatments for anhedonia.

Early-Life Adversities Are Associated With Lower Expected Value Signaling in the Adult Brain

BACKGROUND

Early adverse experiences are assumed to affect fundamental processes of reward learning and decision making. However, computational neuroimaging studies investigating these circuits in the context of adversity are sparse and limited to studies conducted in adolescent samples, leaving the long-term effects unexplored.

METHODS

Using data from a longitudinal birth cohort study (n = 156; 87 female), we investigated associations between adversities and computational markers of reward learning (i.e., expected value, prediction errors). At age 33 years, all participants completed a functional magnetic resonance imaging-based passive avoidance task. Psychopathology measures were collected at the time of functional magnetic resonance imaging investigation and during the COVID-19 pandemic. We applied a principal component analysis to capture common variations across 7 adversity measures. The resulting adversity factors (factor 1: postnatal psychosocial adversities and prenatal maternal smoking; factor 2: prenatal maternal stress and obstetric adversity; factor 3: lower maternal stimulation) were linked with psychopathology and neural responses in the core reward network using multiple regression analysis.

RESULTS

We found that the adversity dimension primarily informed by lower maternal stimulation was linked to lower expected value representation in the right putamen, right nucleus accumbens, and anterior cingulate cortex. Expected value encoding in the right nucleus accumbens further mediated the relationship between this adversity dimension and psychopathology and predicted higher withdrawn symptoms during the COVID-19 pandemic.

CONCLUSIONS

Our results suggested that early adverse experiences in caregiver context might have a long-term disruptive effect on reward learning in reward-related brain regions, which can be associated with suboptimal decision making and thereby may increase the vulnerability of developing psychopathology.

Validation of a touchscreen probabilistic reward task for mice: A reverse-translated assay with cross-species continuity

The Probabilistic Reward Task (PRT) is a laboratory-based technique used to objectively quantify responsivity to reward. The PRT was initially designed to identify reinforcement learning deficits in clinical populations and subsequently was reverse-translated for use in preclinical studies with rats and monkeys. In this task, subjects make visual discriminations and asymmetric probabilistic contingencies are arranged such that correct responses to one stimulus (rich) are reinforced more often than correct responses to the other (lean). Numerous studies have demonstrated that healthy subjects reliably develop a response bias toward the richly rewarded stimulus, whereas humans with anhedonia and laboratory animals with a history of chronic stress exhibit a blunted response bias. This is important because anhedonia, the loss of responsivity to previously rewarding stimuli, is a behavioral phenotype that is a cardinal feature of multiple neuropsychiatric conditions and is without approved pharmacotherapeutic options. To aid in addressing this critical treatment gap, this report describes validation of the first PRT designed for mice, which are a commonly utilized species in preclinical research toward neuropsychiatric medications development. Results reveal orderly psychophysical functions in response to asymmetric probabilistic contingencies in mice, with signal detection outcomes comparable to previous PRT findings in humans, rats, and monkeys. Taken together, such robust cross-species continuity in task performance confirms that the mouse is well-positioned to serve in bidirectional research efforts between human and animal laboratories. These efforts may accelerate the development of treatment options for anhedonia in the different neuropsychiatric conditions in which it is prominent.

Electrophysiological signatures of reward learning in the rodent touchscreen-based Probabilistic Reward Task

Blunted reward learning and reward-related activation within the corticostriatal-midbrain circuitry have been implicated in the pathophysiology of anhedonia and depression. Unfortunately, the search for more efficacious interventions for anhedonic behaviors has been hampered by the use of vastly different preclinical and clinical assays. In a first step in addressing this gap, in the current study, we used event-related potentials and spectral analyses in conjunction with a touchscreen version of the rodent Probabilistic Reward Task (PRT) to identify the electrophysiological signatures of reward learning in rats. We trained 11 rats (5 females and 6 males) on the rodent touchscreen-based PRT and subsequently implanted them with deep electrodes in the anterior cingulate cortex (ACC) and nucleus accumbens (NAc) for local field potentials recordings during the PRT. Behaviorally, the expected responsivity-to-reward profile was observed. At the electrophysiological level, we identified a negative amplitude deflection 250-500 ms after feedback in the ACC and NAc electrodes, as well as power increase in feedback-locked delta (1-5 Hz) and alpha/beta (9-17 Hz) bands in both electrodes for rewarded trials. Using a reverse-translational approach, we identified electrophysiological signatures of reward learning in rats similar to those described in humans. These findings and approaches might provide a useful translational platform to efficiently evaluate novel therapeutics targeting anhedonia.

Pleasure, Reward Value, Prediction Error and Anhedonia

In order to develop effective treatments for anhedonia we need to understand its underlying neurobiological mechanisms. Anhedonia is conceptually strongly linked to reward processing, which involves a variety of cognitive and neural operations. This chapter reviews the evidence for impairments in experiencing hedonic response (pleasure), reward valuation and reward learning based on outcomes (commonly conceptualised in terms of "reward prediction error"). Synthesising behavioural and neuroimaging findings, we examine case-control studies of patients with depression and schizophrenia, including those focusing specifically on anhedonia. Overall, there is reliable evidence that depression and schizophrenia are associated with disrupted reward processing. In contrast to the historical definition of anhedonia, there is surprisingly limited evidence for impairment in the ability to experience pleasure in depression and schizophrenia. There is some evidence that learning about reward and reward prediction error signals are impaired in depression and schizophrenia, but the literature is inconsistent. The strongest evidence is for impairments in the representation of reward value and how this is used to guide action. Future studies would benefit from focusing on impairments in reward processing specifically in anhedonic samples, including transdiagnostically, and from using designs separating different components of reward processing, formulating them in computational terms, and moving beyond cross-sectional designs to provide an assessment of causality.