Risk prediction error coding in orbitofrontal neurons

Reduced neural encoding of utility prediction errors in cocaine addiction

Influential accounts of addiction posit alterations in adaptive behavior driven by deficient dopaminergic prediction errors (PEs), signaling the discrepancy between actual and expected reward. Dopamine neurons encode these error signals in subjective terms, calibrated by individual risk preferences, as "utility" PEs. It remains unclear, however, whether people with drug addiction have PE deficits or their computational source. Here, using an analogous task to prior single-unit studies with known expectancies, we show that fMRI-measured PEs similarly reflect utility PEs. Relative to control participants, people with chronic cocaine addiction demonstrate reduced utility PEs in the dopaminoceptive ventral striatum, with similar trends in orbitofrontal cortex. Dissecting this PE signal into its subcomponent terms attributed these reductions to weaker striatal responses to received reward/utility, whereas suppression of activity with reward expectation was unchanged. These findings support that addiction may fundamentally disrupt PE signaling and reveal an underappreciated role for perceived reward value in this mechanism.

Advances in understanding meso-cortico-limbic-striatal systems mediating risky reward seeking

The risk of an aversive consequence occurring as the result of a reward-seeking action can have a profound effect on subsequent behavior. Such aversive events can be described as punishers, as they decrease the probability that the same action will be produced again in the future and increase the exploration of less risky alternatives. Punishment can involve the omission of an expected rewarding event ("negative" punishment) or the addition of an unpleasant event ("positive" punishment). Although many individuals adaptively navigate situations associated with the risk of negative or positive punishment, those suffering from substance use disorders or behavioral addictions tend to be less able to curtail addictive behaviors despite the aversive consequences associated with them. Here, we discuss the psychological processes underpinning reward seeking despite the risk of negative and positive punishment and consider how behavioral assays in animals have been employed to provide insights into the neural mechanisms underlying addictive disorders. We then review the critical contributions of dopamine signaling to punishment learning and risky reward seeking, and address the roles of interconnected ventral striatal, cortical, and amygdala regions to these processes. We conclude by discussing the ample opportunities for future study to clarify critical gaps in the literature, particularly as related to delineating neural contributions to distinct phases of the risky decision-making process.

Exploring Neural Mechanisms Related to Cognitive Control, Reward, and Affect in Eating Disorders: A Narrative Review of FMRI Studies

Studies using functional magnetic resonance imaging (fMRI) have contributed to our understanding of possible neural abnormalities among individuals with eating disorders. Many of these studies have focused on three domains: 1) cognitive control, 2) reward processing, and 3) affective processing. This review attempts to summarize the recent fMRI findings across these domains among the most well-characterized eating disorders: anorexia nervosa (AN), bulimia nervosa (BN), and binge eating disorder (BED). Though the literature is a bit murky, a few major themes have emerged. Cognitive control systems are affected among individuals across eating disorder diagnoses, but effects seem least pronounced in AN. Specifically, individuals with all eating disorders appear to show decreased prefrontal activation during cognitive control, but there is less evidence in AN linking decreased prefrontal activation with behavior. There is some evidence that the reinforcing value of food is reduced in AN, but individuals with BN and BED show hyperactivation to rewarding food-related stimuli, suggesting the reinforcing value of food may be enhanced. However, more complex reward processing paradigms show that individuals with BN and BED exhibit hypoactivation to reward anticipation and provide mixed results with regards to reward receipt. There are fewer neuroimaging findings related to affective processing, yet behavioral findings suggest affective processing is important in understanding eating disorders. Though the extant literature is complicated, these studies represent a foundation from which to build and provide insight into potential neurobiological mechanisms that may contribute to the pathophysiology of eating disorders.

Nonhuman primate meso-circuitry data: a translational tool to understand brain networks across species

The foundation for understanding brain connections and related psychiatric diseases lies in human and animal circuitry studies. In rodents and nonhuman primates (NHPs), axonal tracing methods provide the ground-truth connectivity information of brain circuits, coupled with the ability to experimentally manipulate them when combined with other methods. In humans, neuroimaging approaches have taken the lead for studying connectivity patterns in vivo and the changes in network profiles associated with disease. To integrate knowledge from animal models and humans, a critical question is how similar the animal brains and circuits are to the humans'. In this review, we demonstrate the use of meso-circuitry information from tracing studies in NHPs to understand common network connections across species. We show that the meso-circuitry information help establish homologies of cortical and striatal regions and fiber pathways between rodents and NHPs, facilitate the translation of connections that are detailed in animal models to humans, and can locate critical hubs in large-scale brain networks. This review combines anatomic studies across animal models and imaging studies across NHPs and humans to provide a more comprehensive understanding of the hard-wired connectivity that underlies neuroimaging-derived brain networks.

Parallel Lemniscal and Non-Lemniscal Sources Control Auditory Responses in the Orbitofrontal Cortex (OFC)

The orbitofrontal cortex (OFC) controls flexible behavior through stimulus value updating based on stimulus outcome associations, allowing seamless navigation in dynamic sensory environments with changing contingencies. Sensory cue driven responses, primarily studied through behavior, exist in the OFC. However, OFC neurons' sensory response properties, particularly auditory, are unknown in the mouse, a genetically tractable animal. We show that mouse OFC single neurons have unique auditory response properties showing pure oddball detection and long timescales of adaptation resulting in stimulus-history dependence. Further, we show that OFC auditory responses are shaped by two parallel sources in the auditory thalamus, lemniscal and non-lemniscal. The latter underlies a large component of the observed oddball detection and additionally controls persistent activity in the OFC through the amygdala. The deviant selectivity can serve as a signal for important changes in the auditory environment. Such signals, if coupled with persistent activity, obtained by disinhibitory control from the non-lemniscal auditory thalamus or amygdala, will allow for associations with a delayed outcome related signal, like reward prediction error, and potentially forms the basis of updating stimulus outcome associations in the OFC. Thus, the baseline sensory responses allow the behavioral requirement-based response modification through relevant inputs from other structures related to reward, punishment, or memory. Thus, alterations in these responses in neurologic disorders can lead to behavioral deficits.

Stress-related changes in financial risk taking: Considering joint effects of cortisol and affect

Many decisions under risk and uncertainty are made under physical or emotional stress. A recent meta-analysis suggested that stress reliably influences risk taking but did not find a relation between single measures of stress such as cortisol and risk taking. One reason for the conflicting findings could be that the influence of stress on risk taking depends not only on physiological but also on psychological stress responses, in particular affective valence. We tested this hypothesis in an exploratory empirical study: Seventy participants worked on a financial risk-taking task. In half of the participants acute stress was induced with a cold pressor task. For all participants we measured cortisol and α-amylase levels, blood pressure, subjective arousal, and affective valence before and after the task. The stress induction increased participants' levels of cortisol, subjective arousal, and systolic blood pressure but did not directly influence negative affect or risky decision making. Examining the interplay between physiological and psychological stress responses, a moderation analysis revealed an interaction between stress induction and affect valence: Negative affect predicted an increase in risk-seeking decision making in the stress condition, but not in the control group. A similar moderation was found with cortisol reactivity, that is, negative affect predicted an increase in risk-seeking decision making in participants with high cortisol reactivity but not in participants with low cortisol reactivity. These results suggest that the effect of stress on risky decision making depends on the interplay of affective valence and cortisol reactivity.

Primate prefrontal neurons signal economic risk derived from the statistics of recent reward experience

Risk derives from the variation of rewards and governs economic decisions, yet how the brain calculates risk from the frequency of experienced events, rather than from explicit risk-descriptive cues, remains unclear. Here, we investigated whether neurons in dorsolateral prefrontal cortex process risk derived from reward experience. Monkeys performed in a probabilistic choice task in which the statistical variance of experienced rewards evolved continually. During these choices, prefrontal neurons signaled the reward-variance associated with specific objects ('object risk') or actions ('action risk'). Crucially, risk was not derived from explicit, risk-descriptive cues but calculated internally from the variance of recently experienced rewards. Support-vector-machine decoding demonstrated accurate neuronal risk discrimination. Within trials, neuronal signals transitioned from experienced reward to risk (risk updating) and from risk to upcoming choice (choice computation). Thus, prefrontal neurons encode the statistical variance of recently experienced rewards, complying with formal decision variables of object risk and action risk.

A connectional hub in the rostral anterior cingulate cortex links areas of emotion and cognitive control

We investigated afferent inputs from all areas in the frontal cortex (FC) to different subregions in the rostral anterior cingulate cortex (rACC). Using retrograde tracing in macaque monkeys, we quantified projection strength by counting retrogradely labeled cells in each FC area. The projection from different FC regions varied across injection sites in strength, following different spatial patterns. Importantly, a site at the rostral end of the cingulate sulcus stood out as having strong inputs from many areas in diverse FC regions. Moreover, it was at the integrative conjunction of three projection trends across sites. This site marks a connectional hub inside the rACC that integrates FC inputs across functional modalities. Tractography with monkey diffusion magnetic resonance imaging (dMRI) located a similar hub region comparable to the tracing result. Applying the same tractography method to human dMRI data, we demonstrated that a similar hub can be located in the human rACC.

Predictive coding of the statistical parameters of uncertain rewards by orbitofrontal neurons

Uncertain reward outcomes are characterised by statistical parameters that capture the numerical values of the underlying probability distributions of reward values, including the expected value, risk (variance) and probability. Here we show coding of an integrated expected value signal by single orbitofrontal neurons in response to visual cues predicting uncertain rewards. Separate subpopulations of orbitofrontal neurons predominantly code the prediction of one statistical parameter with few neurons showing combined coding. These signals are likely combined with subjective value signals to inform learning and decision making under conditions of uncertainty.

Inactivation of Medial Frontal Cortex Changes Risk Preference

Humans and other animals need to make decisions under varying degrees of uncertainty. These decisions are strongly influenced by an individual's risk preference; however, the neuronal circuitry by which risk preference shapes choice is still unclear [1]. Supplementary eye field (SEF), an oculomotor area within primate medial frontal cortex, is thought to be an essential part of the neuronal circuit underlying oculomotor decision making, including decisions under risk [2-5]. Consistent with this view, risk-related action value and monitoring signals have been observed in SEF [6-8]. However, such activity has also been observed in other frontal areas, including orbitofrontal [9-11], cingulate [12-14], and dorsal-lateral frontal cortex [15]. It is thus unknown whether the activity in SEF causally contributes to risky decisions, or whether it is merely a reflection of neural processes in other cortical regions. Here, we tested a causal role of SEF in risky oculomotor choices. We found that SEF inactivation strongly reduced the frequency of risky choices. This reduction was largely due to a reduced attraction to reward uncertainty and high reward gain, but not due to changes in the subjective estimation of reward probability or average expected reward. Moreover, SEF inactivation also led to increased sensitivity to differences between expected and actual reward during free choice. Nevertheless, it did not affect adjustments of decisions based on reward history.

Rule learning enhances structural plasticity of long-range axons in frontal cortex

Rules encompass cue-action-outcome associations used to guide decisions and strategies in a specific context. Subregions of the frontal cortex including the orbitofrontal cortex (OFC) and dorsomedial prefrontal cortex (dmPFC) are implicated in rule learning, although changes in structural connectivity underlying rule learning are poorly understood. We imaged OFC axonal projections to dmPFC during training in a multiple choice foraging task and used a reinforcement learning model to quantify explore–exploit strategy use and prediction error magnitude. Here we show that rule training, but not experience of reward alone, enhances OFC bouton plasticity. Baseline bouton density and gains during training correlate with rule exploitation, while bouton loss correlates with exploration and scales with the magnitude of experienced prediction errors. We conclude that rule learning sculpts frontal cortex interconnectivity and adjusts a thermostat for the explore–exploit balance.

The orbitofrontal cortex is associated with foraging behaviour yet the structural changes underlying such rule-based learning remain unclear. Here, the authors imaged OFC axons throughout a digging-based odour discrimination task and found correlations between the rate of bouton turnover and the behavioural strategies of individual mice.

Dissociable contribution of nucleus accumbens and dorsolateral striatum to the acquisition of risk choice behavior in the rat

While a growing body of research has suggested that the mesocorticolimbic dopamine systems play a key role in decision making under risk, how the nucleus accumbens (NAC) is involved in the acquisition of risk choice behavior remains unclear. This study used a T-maze task to assess risk-based decision making in which the rat was required to assess the risk by choosing to enter either a small and certain reward arm or a large but uncertain reward arm of the maze. The latter option, when chosen, resulted in provision of 2, 4, or 8 sweeten pellets with a probability (p) of 0.5, 0.25, or 0.125, respectively. Thus the latter arm provided three different conditions of reward ratio, compared to the choice of former arm, which always provided 1 pellet with p=1. This risk choice task was then run with the expected value being equality between the binary choice options. The experimental rats first received an excitoneurotoxic lesion affecting either the NAC or the dorsolateral striatum (DLS) and this was followed by post-lesion behavioral examination. The sham lesion control rats acquired a stable risk choice with regard to each reward ratio over a 10-day test. The pattern of choice behavior appeared in risk-seeking when p=0.5 to obtain 2 pellets, and was risk-averse when larger reward resulted in lower p. The NAC lesion significantly disrupted the acquisition of the aforementioned risk choice behavior and apparently shifted the choice into a risk-averse style for all three reward ratios. No such effect was observed in the rats with DLS lesions. Neither the gross motor action nor the discrimination of different reward magnitudes was impaired by the lesions affecting either the NAC or DLS as assessed by an additional experiment. These findings suggest that firstly there is heterogeneity between NAC and DLS with respect to risk-based decision making, and that secondly the NAC is involved and critical to the acquisition of behavioral choice under risk, specially when the expected value of the reward under the two choice options is equal.

Neuronal Reward and Decision Signals: From Theories to Data

Rewards are crucial objects that induce learning, approach behavior, choices, and emotions. Whereas emotions are difficult to investigate in animals, the learning function is mediated by neuronal reward prediction error signals which implement basic constructs of reinforcement learning theory. These signals are found in dopamine neurons, which emit a global reward signal to striatum and frontal cortex, and in specific neurons in striatum, amygdala, and frontal cortex projecting to select neuronal populations. The approach and choice functions involve subjective value, which is objectively assessed by behavioral choices eliciting internal, subjective reward preferences. Utility is the formal mathematical characterization of subjective value and a prime decision variable in economic choice theory. It is coded as utility prediction error by phasic dopamine responses. Utility can incorporate various influences, including risk, delay, effort, and social interaction. Appropriate for formal decision mechanisms, rewards are coded as object value, action value, difference value, and chosen value by specific neurons. Although all reward, reinforcement, and decision variables are theoretical constructs, their neuronal signals constitute measurable physical implementations and as such confirm the validity of these concepts. The neuronal reward signals provide guidance for behavior while constraining the free will to act.

Economic risk coding by single neurons in the orbitofrontal cortex

Risk is a ubiquitous feature of the environment for all organisms. Very few things in life are achieved with absolute certainty. Therefore, it is essential that organisms process risky information efficiently to promote adaptive behaviour and enhance survival. Here we outline a clear definition of economic risk derived from economic theory and focus on two experiments in which we have shown subpopulations of single neurons in the orbitofrontal cortex of rhesus macaques that code either economic risk per se or an error-related risk signal, namely a risk prediction error. These biological risk signals are essential for processing and updating risky information in the environment to contribute to efficient decision making and adaptive behaviour.

Neural processing of reward in adolescent rodents

•

The adolescent brain processes rewards differently than in adults.

•

These differences occur even when behavior is similar between age groups.

•

DS was the locus of substantial developmental differences in reward activity.

•

Surprisingly, differences were not as pronounced in VS.

•

These differences may have implications for adolescent psychiatric vulnerability.

Immaturities in adolescent reward processing are thought to contribute to poor decision making and increased susceptibility to develop addictive and psychiatric disorders. Very little is known; however, about how the adolescent brain processes reward. The current mechanistic theories of reward processing are derived from adult models. Here we review recent research focused on understanding of how the adolescent brain responds to rewards and reward-associated events. A critical aspect of this work is that age-related differences are evident in neuronal processing of reward-related events across multiple brain regions even when adolescent rats demonstrate behavior similar to adults. These include differences in reward processing between adolescent and adult rats in orbitofrontal cortex and dorsal striatum. Surprisingly, minimal age related differences are observed in ventral striatum, which has been a focal point of developmental studies. We go on to discuss the implications of these differences for behavioral traits affected in adolescence, such as impulsivity, risk-taking, and behavioral flexibility. Collectively, this work suggests that reward-evoked neural activity differs as a function of age and that regions such as the dorsal striatum that are not traditionally associated with affective processing in adults may be critical for reward processing and psychiatric vulnerability in adolescents.

Dopamine reward prediction error responses reflect marginal utility

BACKGROUND

Optimal choices require an accurate neuronal representation of economic value. In economics, utility functions are mathematical representations of subjective value that can be constructed from choices under risk. Utility usually exhibits a nonlinear relationship to physical reward value that corresponds to risk attitudes and reflects the increasing or decreasing marginal utility obtained with each additional unit of reward. Accordingly, neuronal reward responses coding utility should robustly reflect this nonlinearity.

RESULTS

In two monkeys, we measured utility as a function of physical reward value from meaningful choices under risk (that adhered to first- and second-order stochastic dominance). The resulting nonlinear utility functions predicted the certainty equivalents for new gambles, indicating that the functions' shapes were meaningful. The monkeys were risk seeking (convex utility function) for low reward and risk avoiding (concave utility function) with higher amounts. Critically, the dopamine prediction error responses at the time of reward itself reflected the nonlinear utility functions measured at the time of choices. In particular, the reward response magnitude depended on the first derivative of the utility function and thus reflected the marginal utility. Furthermore, dopamine responses recorded outside of the task reflected the marginal utility of unpredicted reward. Accordingly, these responses were sufficient to train reinforcement learning models to predict the behaviorally defined expected utility of gambles.

CONCLUSIONS

These data suggest a neuronal manifestation of marginal utility in dopamine neurons and indicate a common neuronal basis for fundamental explanatory constructs in animal learning theory (prediction error) and economic decision theory (marginal utility).