Attenuation of dopamine-modulated prefrontal value signals underlies probabilistic reward learning deficits in old age

Understanding music and aging through the lens of Bayesian inference

Bayesian inference has recently gained momentum in explaining music perception and aging. A fundamental mechanism underlying Bayesian inference is the notion of prediction. This framework could explain how predictions pertaining to musical (melodic, rhythmic, harmonic) structures engender action, emotion, and learning, expanding related concepts of music research, such as musical expectancies, groove, pleasure, and tension. Moreover, a Bayesian perspective of music perception may shed new insights on the beneficial effects of music in aging. Aging could be framed as an optimization process of Bayesian inference. As predictive inferences refine over time, the reliance on consolidated priors increases, while the updating of prior models through Bayesian inference attenuates. This may affect the ability of older adults to estimate uncertainties in their environment, limiting their cognitive and behavioral repertoire. With Bayesian inference as an overarching framework, this review synthesizes the literature on predictive inferences in music and aging, and details how music could be a promising tool in preventive and rehabilitative interventions for older adults through the lens of Bayesian inference.

How does apathy impact exploration-exploitation decision-making in older patients with neurocognitive disorders?

Apathy is a pervasive clinical syndrome in neurocognitive disorders, characterized by a quantitative reduction in goal-directed behaviors. The brain structures involved in the physiopathology of apathy have also been connected to the brain structures involved in probabilistic reward learning in the exploration-exploitation dilemma. This dilemma in question involves the challenge of selecting between a familiar option with a more predictable outcome, and another option whose outcome is uncertain and may yield potentially greater rewards compared to the known option. The aim of this study was to combine experimental procedures and computational modeling to examine whether, in older adults with mild neurocognitive disorders, apathy affects performance in the exploration-exploitation dilemma. Through using a four-armed bandit reinforcement-learning task, we showed that apathetic older adults explored more and performed worse than non-apathetic subjects. Moreover, the mental flexibility assessed by the Trail-making test-B was negatively associated with the percentage of exploration. These results suggest that apathy is characterized by an increased explorative behavior and inefficient decision-making, possibly due to weak mental flexibility to switch toward the exploitation of the more rewarding options. Apathetic participants also took longer to make a choice and failed more often to respond in the allotted time, which could reflect the difficulties in action initiation and selection. In conclusion, the present results suggest that apathy in participants with neurocognitive disorders is associated with specific disturbances in the exploration-exploitation trade-off and sheds light on the disturbances in reward processing in patients with apathy.

Anxiety associated with perceived uncontrollable stress enhances expectations of environmental volatility and impairs reward learning

Unavoidable stress can lead to perceived lack of control and learned helplessness, a risk factor for depression. Avoiding punishment and gaining rewards involve updating the values of actions based on experience. Such updating is however useful only if action values are sufficiently stable, something that a lack of control may impair. We examined whether self-reported stress uncontrollability during the first wave of the COVID-19 pandemic predicted impaired reward-learning. In a preregistered study during the first-wave of the COVID-19 pandemic, we used self-reported measures of depression, anxiety, uncontrollable stress, and COVID-19 risk from 427 online participants to predict performance in a three-armed-bandit probabilistic reward learning task. As hypothesised, uncontrollable stress predicted impaired learning, and a greater proportion of probabilistic errors following negative feedback for correct choices, an effect mediated by state anxiety. A parameter from the best-fitting hidden Markov model that estimates expected beliefs that the identity of the optimal choice will shift across images, mediated effects of state anxiety on probabilistic errors and learning deficits. Our findings show that following uncontrollable stress, anxiety promotes an overly volatile representation of the reward-structure of uncertain environments, impairing reward attainment, which is a potential path to anhedonia in depression.

Effects of Adult Age and Functioning of the Locus Coeruleus Norepinephrinergic System on Reward-Based Learning

Age-related impairments in value representations and updating during decision-making and reward-based learning are often related to age-related attenuation in the catecholamine system such as dopamine (DA) and norepinephrine (NE). However, it is unclear to what extent age-related declines in NE functioning in humans affect reward-based decision-making. We conducted a probabilistic decision-making task and applied a Q-learning model to investigate participants' anticipatory values and value sensitivities. Task-related pupil dilations and locus coeruleus (LC) magnetic resonance imaging (MRI) contrast, which served as a potential window of the LC-NE functions, were assessed in younger and older adults. Results showed that in both choice and feedback phases, younger adults' (N = 42, 22 males) pupil dilations negatively correlated with anticipatory values, indicating uncertainty about outcome probabilities. Uncertainty-evoked pupil dilations in older adults (N = 41, 27 males) were smaller, indicating age-related impairments in value estimation and updating. In both age groups, participants who showed a larger uncertainty-evoked pupil dilation exhibited a higher value sensitivity as reflected in the β parameter of the reinforcement Q-learning model. Furthermore, older adults (N = 34, 29 males) showed a lower LC-MRI contrast than younger adults (N = 25, 15 males). The LC-MRI contrast positively correlated with value sensitivity only in older but not in younger adults. These findings suggest that task-related pupillary responses can reflect age-related deficits in value estimation and updating during reward-based decision-making. Our evidence with the LC-MRI contrast further showed the age-related decline of the LC structure in modulating value representations during reward-based learning.SIGNIFICANCE STATEMENT Age-related impairments in value representation and updating during reward-based learning are associated with declines in the catecholamine modulation with age. However, it is unclear how age-related declines in the LC-NE system may affect reward-based learning. Here, we show that compared with younger adults, older adults exhibited reduced uncertainty-induced pupil dilations, suggesting age-related deficits in value estimation and updating. Older adults showed a lower structural MRI of the LC contrast than younger adults, indicating age-related degeneration of the LC structure. The association between the LC-MRI contrast and value sensitivity was only observed in older adults. Our findings may demonstrate a pioneering model to unravel the role of the LC-NE system in reward-based learning in aging.

Effects of positive and negative social feedback on motivation, evaluative learning, and socio-emotional processing

Social rewards and punishments are strong motivators. Since experimental work has focused on young adults using simplistic feedback, the effects of more naturalistic stimuli on motivation, evaluative learning, and socio-emotional processing with advanced age remain unclear. Therefore, we compared the effects of static (photos) vs dynamic (videos) social feedback in a social incentive delay (SID) task in young (18-35 years) and older adults (50-84 years) with neutral, positive, and negative feedback, on response times (RTs), and assessed the emotional valence of feedback cues and feedback videos. We found that anticipating positive and negative social feedback accelerated RTs regardless of age and without additional effects of video feedback. Furthermore, the results suggest a valence transfer from positive feedback videos to predictive cues in both groups (i.e., evaluative learning). Finally, older adults reported less pronounced negative affect for negative feedback videos, indicating age differences in socio-emotional processing. As such, our findings foster our understanding of the underlying cognitive and emotional aspects involved in the processing of social rewards and punishments.

Thyroid hormone rewires cortical circuits to coordinate body-wide metabolism and exploratory drive

Animals adapt to varying environmental conditions by modifying the function of their internal organs, including the brain. To be adaptive, alterations in behavior must be coordinated with the functional state of organs throughout the body. Here we find that thyroid hormone- a prominent regulator of metabolism in many peripheral organs- activates cell-type specific transcriptional programs in anterior regions of cortex of adult mice via direct activation of thyroid hormone receptors. These programs are enriched for axon-guidance genes in glutamatergic projection neurons, synaptic regulators across both astrocytes and neurons, and pro-myelination factors in oligodendrocytes, suggesting widespread remodeling of cortical circuits. Indeed, whole-cell electrophysiology recordings revealed that thyroid hormone induces local transcriptional programs that rewire cortical neural circuits via pre-synaptic mechanisms, resulting in increased excitatory drive with a concomitant sensitization of recruited inhibition. We find that thyroid hormone bidirectionally regulates innate exploratory behaviors and that the transcriptionally mediated circuit changes in anterior cortex causally promote exploratory decision-making. Thus, thyroid hormone acts directly on adult cerebral cortex to coordinate exploratory behaviors with whole-body metabolic state.

Trait anxiety is associated with hidden state inference during aversive reversal learning

Updating beliefs in changing environments can be driven by gradually adapting expectations or by relying on inferred hidden states (i.e. contexts), and changes therein. Previous work suggests that increased reliance on context could underly fear relapse phenomena that hinder clinical treatment of anxiety disorders. We test whether trait anxiety variations in a healthy population influence how much individuals rely on hidden-state inference. In a Pavlovian learning task, participants observed cues that predicted an upcoming electrical shock with repeatedly changing probability, and were asked to provide expectancy ratings on every trial. We show that trait anxiety is associated with steeper expectation switches after contingency reversals and reduced oddball learning. Furthermore, trait anxiety is related to better fit of a state inference, compared to a gradual learning, model when contingency changes are large. Our findings support previous work suggesting hidden-state inference as a mechanism behind anxiety-related to fear relapse phenomena.

Modulation of Motor Vigor by Expectation of Reward Probability Trial-by-Trial Is Preserved in Healthy Ageing and Parkinson's Disease Patients

Motor improvements, such as faster movement times or increased velocity, have been associated with reward magnitude in deterministic contexts. Yet whether individual inferences on reward probability influence motor vigor dynamically remains undetermined. We investigated how dynamically inferring volatile action-reward contingencies modulated motor performance trial-by-trial. We conducted three studies that coupled a reversal learning paradigm with a motor sequence task and used a validated hierarchical Bayesian model to fit trial-by-trial data. In Study 1, we tested healthy younger [HYA; 37 (24 females)] and older adults [HOA; 37 (17 females)], and medicated Parkinson's disease (PD) patients [20 (7 females)]. We showed that stronger predictions about the tendency of the action-reward contingency led to faster performance tempo, commensurate with movement time, on a trial-by-trial basis without robustly modulating reaction time (RT). Using Bayesian linear mixed models, we demonstrated a similar invigoration effect on performance tempo in HYA, HOA, and PD, despite HOA and PD being slower than HYA. In Study 2 [HYA, 39 (29 females)], we additionally showed that retrospective subjective inference about credit assignment did not contribute to differences in motor vigor effects. Last, Study 3 [HYA, 33 (27 females)] revealed that explicit beliefs about the reward tendency (confidence ratings) modulated performance tempo trial-by-trial. Our study is the first to reveal that the dynamic updating of beliefs about volatile action-reward contingencies positively biases motor performance through faster tempo. We also provide robust evidence for a preserved sensitivity of motor vigor to inferences about the action-reward mapping in aging and medicated PD.SIGNIFICANCE STATEMENT Navigating a world rich in uncertainty relies on updating beliefs about the probability that our actions lead to reward. Here, we investigated how inferring the action-reward contingencies in a volatile environment modulated motor vigor trial-by-trial in healthy younger and older adults, and in Parkinson's disease (PD) patients on medication. We found an association between trial-by-trial predictions about the tendency of the action-reward contingency and performance tempo, with stronger expectations speeding the movement. We additionally provided evidence for a similar sensitivity of performance tempo to the strength of these predictions in all groups. Thus, dynamic beliefs about the changing relationship between actions and their outcome enhanced motor vigor. This positive bias was not compromised by age or Parkinson's disease.

PPARα Signaling: A Candidate Target in Psychiatric Disorder Management

Peroxisome proliferator-activator receptors (PPARs) regulate lipid and glucose metabolism, control inflammatory processes, and modulate several brain functions. Three PPAR isoforms have been identified, PPARα, PPARβ/δ, and PPARγ, which are expressed in different tissues and cell types. Hereinafter, we focus on PPARα involvement in the pathophysiology of neuropsychiatric and neurodegenerative disorders, which is underscored by PPARα localization in neuronal circuits involved in emotion modulation and stress response, and its role in neurodevelopment and neuroinflammation. A multiplicity of downstream pathways modulated by PPARα activation, including glutamatergic neurotransmission, upregulation of brain-derived neurotrophic factor, and neurosteroidogenic effects, encompass mechanisms underlying behavioral regulation. Modulation of dopamine neuronal firing in the ventral tegmental area likely contributes to PPARα effects in depression, anhedonia, and autism spectrum disorder (ASD). Based on robust preclinical evidence and the initial results of clinical studies, future clinical trials should assess the efficacy of PPARα agonists in the treatment of mood and neurodevelopmental disorders, such as depression, schizophrenia, and ASD.

Impaired learning to dissociate advantageous and disadvantageous risky choices in adolescents

Adolescence is characterized by a surge in maladaptive risk-taking behaviors, but whether and how this relates to developmental changes in experience-based learning is largely unknown. In this preregistered study, we addressed this issue using a novel task that allowed us to separate the learning-driven optimization of risky choice behavior over time from overall risk-taking tendencies. Adolescents (12-17 years old) learned to dissociate advantageous from disadvantageous risky choices less well than adults (20-35 years old), and this impairment was stronger in early than mid-late adolescents. Computational modeling revealed that adolescents' suboptimal performance was largely due to an inefficiency in core learning and choice processes. Specifically, adolescents used a simpler, suboptimal, expectation-updating process and a more stochastic choice policy. In addition, the modeling results suggested that adolescents, but not adults, overvalued the highest rewards. Finally, an exploratory latent-mixture model analysis indicated that a substantial proportion of the participants in each age group did not engage in experience-based learning but used a gambler's fallacy strategy, stressing the importance of analyzing individual differences. Our results help understand why adolescents tend to make more, and more persistent, maladaptive risky decisions than adults when the values of these decisions have to be learned from experience.

Multi-scale neural decoding and analysis

Objective.

Complex spatiotemporal neural activity encodes rich information related to behavior and cognition. Conventional research has focused on neural activity acquired using one of many different measurement modalities, each of which provides useful but incomplete assessment of the neural code. Multi-modal techniques can overcome tradeoffs in the spatial and temporal resolution of a single modality to reveal deeper and more comprehensive understanding of system-level neural mechanisms. Uncovering multi-scale dynamics is essential for a mechanistic understanding of brain function and for harnessing neuroscientific insights to develop more effective clinical treatment.

Approach.

We discuss conventional methodologies used for characterizing neural activity at different scales and review contemporary examples of how these approaches have been combined. Then we present our case for integrating activity across multiple scales to benefit from the combined strengths of each approach and elucidate a more holistic understanding of neural processes.

Main results.

We examine various combinations of neural activity at different scales and analytical techniques that can be used to integrate or illuminate information across scales, as well the technologies that enable such exciting studies. We conclude with challenges facing future multi-scale studies, and a discussion of the power and potential of these approaches.

Significance.

This roadmap will lead the readers toward a broad range of multi-scale neural decoding techniques and their benefits over single-modality analyses. This Review article highlights the importance of multi-scale analyses for systematically interrogating complex spatiotemporal mechanisms underlying cognition and behavior.

Involvement of the dopaminergic system in the reward-related behavior of pregabalin

There has been an increase in cases of drug addiction and prescription drug abuse worldwide. Recently, pregabalin abuse has been a focus for many healthcare agencies, as highlighted by epidemiological studies. We previously evaluated the possibility of pregabalin abuse using the conditioned place preference (CPP) paradigm. We observed that a 60 mg/kg dose could induce CPP in mice and that pregabalin-rewarding properties were mediated through glutamate neurotransmission. Notably, the dopaminergic reward circuitry is also known to play a crucial role in medication-seeking behavior. Therefore, this study aimed to explore the possible involvement of dopaminergic receptor-1 in pregabalin-induced CPP. Mice were randomly allocated to receive saline or the dopamine-1 receptor antagonist SKF-83566 (0.03 mg/kg, intraperitoneal). After 30 min, the mice received either saline or pregabalin (60 mg/kg) during the conditioning phase. Among the control groups that received saline or SKF-83566, the time spent in the two conditioning chambers was not significantly altered. However, among the pregabalin-treated group, there was a marked increase in the time spent in the drug-paired chamber compared to the time spent in the vehicle-paired chamber. Notably, blocking dopamine-1 receptors with SKF-83566 completely prevented pregabalin-induced place preference, thus demonstrating the engagement of the dopaminergic system in pregabalin-induced reward-related behavior.

Role of dopamine and gray matter density in aging effects and individual differences of functional connectomes

With increasing age, functional connectomes become dissimilar across normal individuals, reflecting heterogenous aging effects on functional connectivity (FC). We investigated the distribution of these effects across the connectome and their relationship with age-related differences in dopamine (DA) D1 receptor availability and gray matter density (GMD). With this aim, we determined aging effects on mean and interindividual variance of FC using fMRI in 30 younger and 30 older healthy subjects and mapped the contribution of each connection to the patterns of age-related similarity loss. Aging effects on mean FC accounted mainly for the dissimilarity between connectomes of younger and older adults, and were related, across brain regions, to aging effects on DA D1 receptor availability. Aging effects on the variance of FC indicated a global increase in variance with advancing age, explained connectome dissimilarity among older subjects and were related to aging effects on variance of GMD. The relationship between aging and the similarity of connectomes can thus be partly explained by age differences in DA modulation and gray matter structure.

Corticostriatal White Matter Integrity and Dopamine D1 Receptor Availability Predict Age Differences in Prefrontal Value Signaling during Reward Learning

Probabilistic reward learning reflects the ability to adapt choices based on probabilistic feedback. The dopaminergically innervated corticostriatal circuit in the brain plays an important role in supporting successful probabilistic reward learning. Several components of the corticostriatal circuit deteriorate with age, as it does probabilistic reward learning. We showed previously that D1 receptor availability in NAcc predicts the strength of anticipatory value signaling in vmPFC, a neural correlate of probabilistic learning that is attenuated in older participants and predicts probabilistic reward learning performance. We investigated how white matter integrity in the pathway between nucleus accumbens (NAcc) and ventromedial prefrontal cortex (vmPFC) relates to the strength of anticipatory value signaling in vmPFC in younger and older participants. We found that in a sample of 22 old and 23 young participants, fractional anisotropy in the pathway between NAcc and vmPFC predicted the strength of value signaling in vmPFC independently from D1 receptor availability in NAcc. These findings provide tentative evidence that integrity in the dopaminergic and white matter pathways of corticostriatal circuitry supports the expression of value signaling in vmPFC which supports reward learning, however, the limited sample size calls for independent replication. These and future findings could add to the improved understanding of how corticostriatal integrity contributes to reward learning ability.

Motivation and sensitivity to monetary reward in late-life insomnia: moderating role of sex and the inflammatory marker CRP

Insomnia is a well-established risk factor for late-life depression, yet the intermediary mechanisms are not known. One plausible mechanism is dysregulation of the reward system, a common feature of depression. The main objective of the current study was to determine whether late-life insomnia is associated with reduced motivation and reduced sensitivity for monetary reward. Secondary exploratory objectives were to test for sex-specific effects and whether elevated inflammation potentiated these associations. Nondepressed community dwelling older adults (n = 104; aged 60-80) who either met (n = 31) or did not meet (n = 73) criteria for insomnia disorder as assessed by the Structured Clinical Interview for DSM-5 completed the Effort Expenditure for Rewards Task and provided blood samples for the assessment of C-reactive protein (CRP). Older adults with late-life insomnia showed reduced reward motivation 95% CI [-0.955, -0.569] and reduced reward sensitivity 95% CI [-0.430, -0.075] relative to comparison controls. In secondary exploratory analyses, late-life insomnia was associated with reduced motivation to a greater degree in males than in females 95% CI [0.072, 0.775], particularly when CRP was also elevated 95% CI [0.672, 1.551]. Late-life insomnia is associated with reduced motivation and sensitivity for monetary reward, which suggests insomnia may confer risk for late-life depression by dysregulation of reward mechanisms. Exploratory analyses suggest that older males with insomnia and elevated CRP may be particularly vulnerable to deficits in reward motivation. Although in need of replication and further study, results suggest that interventions that target insomnia or deficits in reward processing may mitigate the risk of depression in nondepressed older adults, especially older males with insomnia.

Associations between aversive learning processes and transdiagnostic psychiatric symptoms in a general population sample

Symptom expression in psychiatric conditions is often linked to altered threat perception, however how computational mechanisms that support aversive learning relate to specific psychiatric symptoms remains undetermined. We answer this question using an online game-based aversive learning task together with measures of common psychiatric symptoms in 400 subjects. We show that physiological symptoms of anxiety and a transdiagnostic compulsivity-related factor are associated with enhanced safety learning, as measured using a probabilistic computational model, while trait cognitive anxiety symptoms are associated with enhanced learning from danger. We use data-driven partial least squares regression to identify two separable components across behavioural and questionnaire data: one linking enhanced safety learning and lower estimated uncertainty to physiological anxiety, compulsivity, and impulsivity; the other linking enhanced threat learning and heightened uncertainty estimation to symptoms of depression and social anxiety. Our findings implicate aversive learning processes in the expression of psychiatric symptoms that transcend diagnostic boundaries.

Dopamine D1 receptor availability is not associated with delusional ideation measures of psychosis proneness

The dopamine D1 receptor (D1R) is thought to play a role in psychosis and schizophrenia, however positron emission tomography studies comparing patients and controls have been inconsistent. To circumvent some of the limitations of clinical studies, such as antipsychotic exposure, an alternative approach is to examine subclinical psychotic symptoms within the general population, i.e. psychosis proneness traits. In this study, we investigated whether D1R availability is associated with delusional ideation in healthy controls, in four experiments, using [¹¹C]SCH23390 PET (n = 76) and psychometric questionnaires (n = 217). We performed exploratory analyses, direct self-replication, and confirmatory analyses using Bayesian statistical modelling. Collectively, we found strong evidence that there is little to no linear association between delusional ideation and D1R. If hypothesised changes in D1R in drug-naive psychosis patients can be confirmed, our results suggest that they may either occur at disease onset, or that they are associated with specific aspects of psychosis other than delusional ideation.

Age-related variability in decision-making: Insights from neurochemistry

Despite dopamine's significant role in models of value-based decision-making and findings demonstrating loss of dopamine function in aging, evidence of systematic changes in decision-making over the life span remains elusive. Previous studies attempting to resolve the neural basis of age-related alteration in decision-making have typically focused on physical age, which can be a poor proxy for age-related effects on neural systems. There is growing appreciation that aging has heterogeneous effects on distinct components of the dopamine system within subject in addition to substantial variability between subjects. We propose that some of the conflicting findings in age-related effects on decision-making may be reconciled if we can observe the underlying dopamine components within individuals. This can be achieved by incorporating in vivo imaging techniques including positron emission tomography (PET) and neuromelanin-sensitive MR. Further, we discuss how affective factors may contribute to individual differences in decision-making performance among older adults. Specifically, we propose that age-related shifts in affective attention ("positivity effect") can, in some cases, counteract the impact of altered dopamine function on specific decision-making processes, contributing to variability in findings. In an effort to provide clarity to the field and advance productive hypothesis testing, we propose ways in which in vivo dopamine imaging can be leveraged to disambiguate dopaminergic influences on decision-making, and suggest strategies for assessing individual differences in the contribution of affective attentional focus.

Age-related cognitive decline in baboons: modeling the prodromal phase of Alzheimer's disease and related dementias

The aging of brain cells and synaptic loss are the major underlying pathophysiological processes contributing to the progressive decline in cognitive functions and Alzheimer’s disease. The difference in cognitive performances observed between adult and aged subjects across species highlights the decline of brain systems with age. The inflection point in age-related cognitive decline is important for our understanding of the pathophysiology of neurodegenerative diseases and for timing therapeutic interventions. Humans and nonhuman primates share many similarities including age-dependent changes in gene expression and decline in neural and immune functions. Given these evolutionary conserved organ systems, complex human-like behavioral and age-dependent changes may be modeled and monitored longitudinally in nonhuman primates. We integrated three clinically relevant outcome measures to investigate the effect of age on cognition, motor function and diurnal activity in aged baboons. We provide evidence of a naturally-occurring age-dependent precipitous decline in movement planning, in learning novel tasks, in simple discrimination and in motivation. These results suggest that baboons aged ~20 years (equivalent to ~60 year old humans) may offer a relevant model for the prodromal phase of Alzheimer’s disease and related dementias to investigate mechanisms involved in the precipitous decline in cognitive functions and to develop early therapeutic interventions

Activation of D1 receptors affects human reactivity and flexibility to valued cues

Reward-predicting cues motivate goal-directed behavior, but in unstable environments humans must also be able to flexibly update cue-reward associations. While the capacity of reward cues to trigger motivation ('reactivity') as well as flexibility in cue-reward associations have been linked to the neurotransmitter dopamine in humans, the specific contribution of the dopamine D1 receptor family to these behaviors remained elusive. To fill this gap, we conducted a randomized, placebo-controlled, double-blind pharmacological study testing the impact of three different doses of a novel D1 agonist (relative to placebo) on reactivity to reward-predicting cues (Pavlovian-to-instrumental transfer) and flexibility of cue-outcome associations (reversal learning). We observed that the impact of the D1 agonist crucially depended on baseline working memory functioning, which has been identified as a proxy for baseline dopamine synthesis capacity. Specifically, increasing D1 receptor stimulation strengthened Pavlovian-to-instrumental transfer in individuals with high baseline working memory capacity. In contrast, higher doses of the D1 agonist improved reversal learning only in individuals with low baseline working memory functioning. Our findings suggest a crucial and baseline-dependent role of D1 receptor activation in controlling both cue reactivity and the flexibility of cue-reward associations.

Variability in Action Selection Relates to Striatal Dopamine 2/3 Receptor Availability in Humans: A PET Neuroimaging Study Using Reinforcement Learning and Active Inference Models

Choosing actions that result in advantageous outcomes is a fundamental function of nervous systems. All computational decision-making models contain a mechanism that controls the variability of (or confidence in) action selection, but its neural implementation is unclear-especially in humans. We investigated this mechanism using two influential decision-making frameworks: active inference (AI) and reinforcement learning (RL). In AI, the precision (inverse variance) of beliefs about policies controls action selection variability-similar to decision 'noise' parameters in RL-and is thought to be encoded by striatal dopamine signaling. We tested this hypothesis by administering a 'go/no-go' task to 75 healthy participants, and measuring striatal dopamine 2/3 receptor (D2/3R) availability in a subset (n = 25) using [11C]-(+)-PHNO positron emission tomography. In behavioral model comparison, RL performed best across the whole group but AI performed best in participants performing above chance levels. Limbic striatal D2/3R availability had linear relationships with AI policy precision (P = 0.029) as well as with RL irreducible decision 'noise' (P = 0.020), and this relationship with D2/3R availability was confirmed with a 'decision stochasticity' factor that aggregated across both models (P = 0.0006). These findings are consistent with occupancy of inhibitory striatal D2/3Rs decreasing the variability of action selection in humans.

Mesolimbic dopamine D2 receptors and neural representations of subjective value

The process by which the value of delayed rewards is discounted varies from person to person. It has been suggested that these individual differences in subjective valuation of delayed rewards are supported by mesolimbic dopamine D2-like receptors (D2Rs) in the ventral striatum. However, no study to date has documented an association between direct measures of dopamine receptors and neural representations of subjective value in humans. Here, we examined whether individual differences in D2R availability were related to neural subjective value signals during decision making. Human participants completed a monetary delay discounting task during an fMRI scan and on a separate visit completed a PET scan with the high affinity D2R tracer [18 F]fallypride. Region-of-interest analyses revealed that D2R availability in the ventral striatum was positively correlated with subjective value-related activity in the ventromedial prefrontal cortex and midbrain but not with choice behavior. Whole-brain analyses revealed a positive correlation between ventral striatum D2R availability and subjective value-related activity in the left inferior frontal gyrus and superior insula. These findings identify a link between a direct measure of mesolimbic dopamine function and subjective value representation in humans and suggest a mechanism by which individuals vary in neural representation of discounted subjective value.

A computational account of threat-related attentional bias

Visual selective attention acts as a filter on perceptual information, facilitating learning and inference about important events in an agent's environment. A role for visual attention in reward-based decisions has previously been demonstrated, but it remains unclear how visual attention is recruited during aversive learning, particularly when learning about multiple stimuli concurrently. This question is of particular importance in psychopathology, where enhanced attention to threat is a putative feature of pathological anxiety. Using an aversive reversal learning task that required subjects to learn, and exploit, predictions about multiple stimuli, we show that the allocation of visual attention is influenced significantly by aversive value but not by uncertainty. Moreover, this relationship is bidirectional in that attention biases value updates for attended stimuli, resulting in heightened value estimates. Our findings have implications for understanding biased attention in psychopathology and support a role for learning in the expression of threat-related attentional biases in anxiety.

D1-Dopamine Receptor Availability in First-Episode Neuroleptic Naive Psychosis Patients

BACKGROUND

Positron emission tomography studies examining differences in D1-dopamine receptor binding between control subjects and patients with schizophrenia have been inconsistent, reporting higher, lower, and no difference in the frontal cortex. Exposure to antipsychotic medication has been suggested to be a likely source of this heterogeneity, and thus there is a need for studies of patients at early stages of the disorder who have not been exposed to such drugs.

METHODS

Here, we compared 17 healthy control subjects and 18 first-episode neuroleptic naive patients with schizophrenia or schizophreniform psychosis using positron emission tomography and the D1-dopamine receptor radioligand [11C]SCH23390.

RESULTS

We observed a statistically significant difference in the dorsolateral prefrontal cortex. Contrary to our expectations, patients had less D1-dopamine receptor availability with a moderate effect size. In a Bayesian analysis, we show that the data are over 50 times more likely to have occurred under the decrease as opposed to the increase hypothesis. This effect was not global, as our analysis showed that the null hypothesis was preferred over either hypothesis in the striatum.

CONCLUSIONS

This investigation represents the largest single sample of neuroleptic-naive patients examined for D1-dopamine receptor availability using PET and suggests a reduction of prefrontal D1-dopamine receptor density in the pathophysiology of schizophrenia. However, further work will be required to reach a consensus.

Dorsal striatal dopamine D1 receptor availability predicts an instrumental bias in action learning

The brain’s dopaminergic pathways are crucially important for adaptive behavior. They are thought to enable us to approach rewards and stay away from punishments. During learning, dopaminergic reward prediction errors are thought to reinforce previously rewarded actions, so they become easier to repeat. This dopaminergic activity could lead to a systematic bias by which rewarded actions are more readily learned than rewarded inactions. We present two findings. First, dopamine receptors in cortex, dorsal striatum, and nucleus accumbens provide distinct sources of variance in the human brain. Second, the boost in an individual’s learning rate from previously rewarded actions is dependent on the dopamine receptor density in dorsal striatum, a central structure in the dopaminergic circuit.

Learning to act to obtain reward and inhibit to avoid punishment is easier compared with learning the opposite contingencies. This coupling of action and valence is often thought of as a Pavlovian bias, although recent research has shown it may also emerge through instrumental mechanisms. We measured this learning bias with a rewarded go/no-go task in 60 adults of different ages. Using computational modeling, we characterized the bias as being instrumental. To assess the role of endogenous dopamine (DA) in the expression of this bias, we quantified DA D1 receptor availability using positron emission tomography (PET) with the radioligand [¹¹C]SCH23390. Using principal-component analysis on the binding potentials in a number of cortical and striatal regions of interest, we demonstrated that cortical, dorsal striatal, and ventral striatal areas provide independent sources of variance in DA D1 receptor availability. Interindividual variation in the dorsal striatal component was related to the strength of the instrumental bias during learning. These data suggest at least three anatomical sources of variance in DA D1 receptor availability separable using PET in humans, and we provide evidence that human dorsal striatal DA D1 receptors are involved in the modulation of instrumental learning biases.

Dopamine D2/3 Binding Potential Modulates Neural Signatures of Working Memory in a Load-Dependent Fashion

Dopamine (DA) modulates corticostriatal connections. Studies in which imaging of the DA system is integrated with functional imaging during cognitive performance have yielded mixed findings. Some work has shown a link between striatal DA (measured by PET) and fMRI activations, whereas others have failed to observe such a relationship. One possible reason for these discrepant findings is differences in task demands, such that a more demanding task with greater prefrontal activations may yield a stronger association with DA. Moreover, a potential DA–BOLD association may be modulated by task performance. We studied 155 (104 normal-performing and 51 low-performing) healthy older adults (43% females) who underwent fMRI scanning while performing a working memory (WM) n-back task along with DA D_2/3 PET assessment using [¹¹C]raclopride. Using multivariate partial-least-squares analysis, we observed a significant pattern revealing positive associations of striatal as well as extrastriatal DA D_2/3 receptors to BOLD response in the thalamo–striatal–cortical circuit, which supports WM functioning. Critically, the DA–BOLD association in normal-performing, but not low-performing, individuals was expressed in a load-dependent fashion, with stronger associations during 3-back than 1-/2-back conditions. Moreover, normal-performing adults expressing upregulated BOLD in response to increasing task demands showed a stronger DA–BOLD association during 3-back, whereas low-performing individuals expressed a stronger association during 2-back conditions. This pattern suggests a nonlinear DA–BOLD performance association, with the strongest link at the maximum capacity level. Together, our results suggest that DA may have a stronger impact on functional brain responses during more demanding cognitive tasks.

SIGNIFICANCE STATEMENT Dopamine (DA) is a major neuromodulator in the CNS and plays a key role in several cognitive processes via modulating the blood oxygenation level-dependent (BOLD) signal. Some studies have shown a link between DA and BOLD, whereas others have failed to observe such a relationship. A possible reason for the discrepancy is differences in task demands, such that a more demanding task with greater prefrontal activations may yield a stronger association with DA. We examined the relationship of DA to BOLD response during working memory under three load conditions and found that the DA–BOLD association is expressed in a load-dependent fashion. These findings may help explain the disproportionate impairment evident in more effortful cognitive tasks in normal aging and in those suffering dopamine-dependent neurodegenerative diseases (e.g., Parkinson's disease).

Older adults fail to form stable task representations during model-based reversal inference

Older adults struggle in dealing with changeable and uncertain environments across several cognitive domains. This has been attributed to difficulties in forming adequate task representations that help navigate uncertain environments. Here, we investigate how, in older adults, inadequate task representations impact on model-based reversal learning. We combined computational modeling and pupillometry during a novel model-based reversal learning task, which allowed us to isolate the relevance of task representations at feedback evaluation. We find that older adults overestimate the changeability of task states and consequently are less able to converge on unequivocal task representations through learning. Pupillometric measures and behavioral data show that these unreliable task representations in older adults manifest as a reduced ability to focus on feedback that is relevant for updating task representations, and as a reduced metacognitive awareness in the accuracy of their actions. Instead, the data suggested older adults' choice behavior was more consistent with a guidance by uninformative feedback properties such as outcome valence. Our study highlights that an inability to form adequate task representations may be a crucial factor underlying older adults' impaired model-based inference.

•

Older adults overestimate the changeability of task states in uncertain environments.

•

Unreliable task representations impact model updating in older adults.

•

Older adults focus on outcome valence rather than model updating during feedback.

•

Older adults show reduced metacognitive awareness of their decision-making accuracy.

•

Pupil diameter encodes model updating and not surprise in young and older adults.

PET radioligands for the dopamine D1-receptor: Application in psychiatric disorders

The dopamine (DA) system is considered to be centrally involved in the pathophysiology of several major psychiatric disorders. Using positron emission tomography (PET), aberrations in dopamine D2/D3-receptors (D2-R) levels and uptake of the DA precursor FDOPA have been shown for schizophrenia, substance abuse and depression. Radioligands for the dopamine D1-receptor (D1-R) have been available for more than three decades, however this receptor subtype has received much less attention in psychiatry research. Here, studies investigating D1-R in psychiatric patients in comparison to healthy control subjects are summarized. Although small sample sizes, medication effects and heterogeneous methods of quantification limit the conclusions that can be drawn, the data is suggestive of higher levels of cortical D1-R in drug naïve patients with psychosis, and lower D1-R in patients with affective disorders. Data sharing and reanalysis using harmonized methodology are important next steps towards clarifying the role of D1-R in these disorders.

Developmental Changes in Learning: Computational Mechanisms and Social Influences

Our ability to learn from the outcomes of our actions and to adapt our decisions accordingly changes over the course of the human lifespan. In recent years, there has been an increasing interest in using computational models to understand developmental changes in learning and decision-making. Moreover, extensions of these models are currently applied to study socio-emotional influences on learning in different age groups, a topic that is of great relevance for applications in education and health psychology. In this article, we aim to provide an introduction to basic ideas underlying computational models of reinforcement learning and focus on parameters and model variants that might be of interest to developmental scientists. We then highlight recent attempts to use reinforcement learning models to study the influence of social information on learning across development. The aim of this review is to illustrate how computational models can be applied in developmental science, what they can add to our understanding of developmental mechanisms and how they can be used to bridge the gap between psychological and neurobiological theories of development.