Deconstructing value-based decision making via temporally selective manipulation of neural activity: Insights from rodent models

Lateral Orbitofrontal Cortex Encodes Presence of Risk and Subjective Risk Preference During Decision-Making

Adaptive decision-making requires consideration of objective risks and rewards associated with each option, as well as subjective preference for risky/safe alternatives. Inaccurate risk/reward estimations can engender excessive risk-taking, a central trait in many psychiatric disorders. The lateral orbitofrontal cortex (lOFC) has been linked to many disorders associated with excessively risky behavior and is ideally situated to mediate risky decision-making. Here, we used single-unit electrophysiology to measure neuronal activity from lOFC of freely moving rats performing in a punishment-based risky decision-making task. Subjects chose between a small, safe reward and a large reward associated with either 0% or 50% risk of concurrent punishment. lOFC activity repeatedly encoded current risk in the environment throughout the decision-making sequence, signaling risk before, during, and after a choice. In addition, lOFC encoded reward magnitude, although this information was only evident during action selection. A Random Forest classifier successfully used neural data accurately to predict the risk of punishment in any given trial, and the ability to predict choice via lOFC activity differentiated between and risk-preferring and risk-averse rats. Finally, risk preferring subjects demonstrated reduced lOFC encoding of risk and increased encoding of reward magnitude. These findings suggest lOFC may serve as a central decision-making hub in which external, environmental information converges with internal, subjective information to guide decision-making in the face of punishment risk.

Circuit and Cell-Specific Contributions to Decision Making Involving Risk of Explicit Punishment in Male and Female Rats

Decision making is a complex cognitive process that recruits a distributed network of brain regions, including the basolateral amygdala (BLA) and nucleus accumbens shell (NAcSh). Recent work suggests that communication between these structures, as well as activity of cells expressing dopamine (DA) D2 receptors (D2R) in the NAcSh, are necessary for some forms of decision making; however, the contributions of this circuit and cell population during decision making under risk of punishment are unknown. The current experiments addressed this question using circuit-specific and cell type-specific optogenetic approaches in rats during a decision making task involving risk of punishment. In experiment 1, Long-Evans rats received intra-BLA injections of halorhodopsin or mCherry (control) and in experiment 2, D2-Cre transgenic rats received intra-NAcSh injections of Cre-dependent halorhodopsin or mCherry. Optic fibers were implanted in the NAcSh in both experiments. Following training in the decision making task, BLA→NAcSh or D2R-expressing neurons were optogenetically inhibited during different phases of the decision process. Inhibition of the BLA→NAcSh during deliberation (the time between trial initiation and choice) increased preference for the large, risky reward (increased risk taking). Similarly, inhibition during delivery of the large, punished reward increased risk taking, but only in males. Inhibition of D2R-expressing neurons in the NAcSh during deliberation increased risk taking. In contrast, inhibition of these neurons during delivery of the small, safe reward decreased risk taking. These findings extend our knowledge of the neural dynamics of risk taking, revealing sex-dependent circuit recruitment and dissociable activity of selective cell populations during decision making.SIGNIFICANCE STATEMENT Until recently, the ability to dissect the neural substrates of decision making involving risk of punishment (risk taking) in a circuit-specific and cell-specific manner has been limited by the tools available for use in rats. Here, we leveraged the temporal precision of optogenetics, together with transgenic rats, to probe contributions of a specific circuit and cell population to different phases of risk-based decision making. Our findings reveal basolateral amygdala (BLA)→nucleus accumbens shell (NAcSh) is involved in evaluation of punished rewards in a sex-dependent manner. Further, NAcSh D2 receptor (D2R)-expressing neurons make unique contributions to risk taking that vary across the decision making process. These findings advance our understanding of the neural principles of decision making and provide insight into how risk taking may become compromised in neuropsychiatric diseases.

Temporal Dynamics Underlying Prelimbic Prefrontal Cortical Regulation of Action Selection and Outcome Evaluation during Risk/Reward Decision-Making

Risk/reward decision-making is a dynamic process that includes periods of deliberation before action selection and evaluation of the action outcomes that bias subsequent choices. Inactivation of the prelimbic (PL) cortex has revealed its integral role in updating decision biases in the face of changes in probabilistic reward contingencies, yet how phasic PL signals during different phases of the decision process influence choice remains unclear. We used temporally specific optogenetic inhibition to selectively disrupt PL activity coinciding with action selection and outcome phases to examine how these signals influence choice. Male rats expressing the inhibitory opsin eArchT within PL excitatory neurons were well trained on a probabilistic discounting task, entailing choice between small/certain versus large/risky rewards, the probability of which varied over a session (50-12.5%). During testing, brief light pulses suppressed PL activity before choice or after different outcomes. Prechoice suppression reduced bias toward more preferred/higher utility options and disrupted how recent outcomes influenced subsequent choice. Inhibition during risky losses induced a similar profile, but here, the impact of reward omissions were either amplified or diminished, relative to the context of the estimated profitability of the risky option. Inhibition during large or small reward receipt reduced risky choice when this option was more profitable, suggesting these signals can both reinforce rewarded risky choices and also act as a relative value comparator signal that augments incentive for larger rewards. These findings reveal multifaceted contributions by the PL in implementing decisions and integrating action-outcome feedback to assign context to the decision space.SIGNIFICANCE STATEMENT The PL prefrontal cortex plays an integral role in guiding risk/reward decisions, but how activity in this region during different phases of the decision process influences choice is unclear. By using temporally specific optogenetic manipulations of this activity, the present study unveiled previously uncharacterized and differential contributions by PL in implementing decision policies and how evaluation of decision outcomes shape subsequent choice. These findings provide novel insight into the dynamic processes engaged by the PL that underlie action selection in situations involving reward uncertainty that may aid in understanding the mechanism underlying normal and aberrant decision-making processes.

Circuit and cell-specific contributions to decision making involving risk of explicit punishment in male and female rats

Decision making is a complex cognitive process that recruits a distributed network of brain regions, including the basolateral amygdala (BLA) and nucleus accumbens shell (NAcSh). Recent work suggests that communication between these structures, as well as activity of cells expressing dopamine D2 receptors (D2R) in the NAcSh, are necessary for some forms of decision making; however, the contributions of this circuit and cell population during decision making under risk of punishment are unknown. The current experiments addressed this question using circuit- and cell type-specific optogenetic approaches in rats during a decision-making task involving risk of punishment. In Experiment 1, Long-Evans rats received intra-BLA injections of halorhodopsin or mCherry (control) and in Experiment 2, D2-Cre transgenic rats received intra-NAcSh injections of Cre-dependent halorhodopsin or mCherry. Optic fibers were implanted in the NAcSh in both experiments. Following training in the decision-making task, BLA→NAcSh or D2R-expressing neurons were optogenetically inhibited during different phases of the decision process. Inhibition of the BLA→NAcSh during deliberation (the time between trial initiation and choice) increased choice of the large, risky reward (increased risk taking). Similarly, inhibition during delivery of the large, punished reward increased risk taking, but only in males. Inhibition of D2R-expressing neurons in the NAcSh during deliberation increased risk taking. In contrast, inhibition of these neurons during delivery of the small, safe reward decreased risk taking. These findings extend our knowledge of the neural dynamics of risk taking, revealing sex-dependent circuit recruitment and dissociable activity of selective cell populations during decision making.

Age-Related Changes in Risky Decision Making and Associated Neural Circuitry in a Rat Model

Altered decision making at advanced ages can have a significant impact on an individual's quality of life and the ability to maintain personal independence. Relative to young adults, older adults make less impulsive and less risky choices; although these changes in decision making could be considered beneficial, they can also lead to choices with potentially negative consequences (e.g., avoidance of medical procedures). Rodent models of decision making have been invaluable for dissecting cognitive and neurobiological mechanisms that contribute to age-related changes in decision making, but they have predominantly used costs related to timing or probability of reward delivery and have not considered other equally important costs, such as the risk of adverse consequences. The current study therefore used a rat model of decision making involving risk of explicit punishment to examine age-related changes in this form of choice behavior in male rats, and to identify potential cognitive and neurobiological mechanisms that contribute to these changes. Relative to young rats, aged rats displayed greater risk aversion, which was not attributable to reduced motivation for food, changes in shock sensitivity, or impaired cognitive flexibility. Functional MRI analyses revealed that, overall, functional connectivity was greater in aged rats compared with young rats, particularly among brain regions implicated in risky decision making such as basolateral amygdala, orbitofrontal cortex, and ventral tegmental area. Collectively, these findings are consistent with greater risk aversion found in older humans, and reveal age-related changes in brain connectivity.

Prenatal dexamethasone exposure alters effort decision making and triggers nucleus accumbens and anterior cingulate cortex functional changes in male rats

Daily, individuals select actions based on cost-benefit to allocate resources into goal-directed actions. Different brain regions coordinate this complex decision, including the nucleus accumbens (NAc), anterior cingulate cortex (ACC), and ventral tegmental area (VTA). In utero exposure to synthetic glucocorticoids (iuGC), such as dexamethasone, triggers prominent motivation deficits but the impact of this exposure in the ACC-NAc and/or ACC-VTA circuits is unknown. Here, we show that iuGC exposure causes decreased motivation for natural rewards (food) and impaired effort-based decision-making. Importantly, reduced neuronal activation (number of c-fos⁺ neurons) was observed in the NAc core and ACC of iuGC rats in comparison to CTR rats after performing the effort-based decision-making task. In addition, iuGC treatment led to increased NAc and ACC basal neuronal activity. Electrophysiological recordings during optogenetic modulation of ACC terminals in the NAc revealed that the ACC-NAc circuit is dysfunctional in iuGC animals. These data suggest that iuGC animals present motivational and effort-based decision-making deficits that can be associated with the observed ACC-NAc dysfunction.

Regulation of sex differences in risk-based decision making by gonadal hormones: Insights from rodent models

Men and women differ in their ability to evaluate options that vary in their rewards and the risks that are associated with these outcomes. Most studies have shown that women are more risk averse than men and that gonadal hormones significantly contribute to this sex difference. Gonadal hormones can influence risk-based decision making (i.e., risk taking) by modulating the neurobiological substrates underlying this cognitive process. Indeed, estradiol, progesterone and testosterone modulate activity in the prefrontal cortex, amygdala and nucleus accumbens associated with reward and risk-related information. The use of animal models of decision making has advanced our understanding of the intersection between the behavioral, neural and hormonal mechanisms underlying sex differences in risk taking. This review will outline the current state of this literature, identify the current gaps in knowledge and suggest the neurobiological mechanisms by which hormones regulate risky decision making. Collectively, this knowledge can be used to understand the potential consequences of significant hormonal changes, whether endogenously or exogenously induced, on risk-based decision making as well as the neuroendocrinological basis of neuropsychiatric diseases that are characterized by impaired risk taking, such as substance use disorder and schizophrenia.

Compulsive alcohol drinking in rodents is associated with altered representations of behavioral control and seeking in dorsal medial prefrontal cortex

A key feature of compulsive alcohol drinking is continuing to drink despite negative consequences. To examine the changes in neural activity that underlie this behavior, compulsive alcohol drinking was assessed in a validated rodent model of heritable risk for excessive drinking (alcohol preferring (P) rats). Neural activity was measured in dorsal medial prefrontal cortex (dmPFC-a brain region involved in maladaptive decision-making) and assessed via change point analyses and novel principal component analyses. Neural population representations of specific decision-making variables were measured to determine how they were altered in animals that drink alcohol compulsively. Compulsive animals showed weakened representations of behavioral control signals, but strengthened representations of alcohol seeking-related signals. Finally, chemogenetic-based excitation of dmPFC prevented escalation of compulsive alcohol drinking. Collectively, these data indicate that compulsive alcohol drinking in rats is associated with alterations in dmPFC neural activity that underlie diminished behavioral control and enhanced seeking.

Examining the reliability and validity of two versions of the Effort-Expenditure for Rewards Task (EEfRT)

The Effort-Expenditure for Rewards Task (EEfRT) has gained validity evidence from several studies. However, various modifications have been applied to the original version, which have never been compared systematically. In Study 1, we tested 120 healthy participants to directly compare two versions of the EEfRT. In Study 2, we tested a larger sample of 394 healthy participants to further examine the original EEfRT. We replicated the split-half reliability of both task versions. However, self-reported personality traits (e.g., trait BAS) correlated with only some task performance parameters in Study 1, which did not replicate for the original EEfRT in Study 2. Our results indicate complex and sometimes inconsistent relations between different personality traits, task properties, and reward attributes.

Relationship of the balloon analog risk task to neurocognitive impairment differs by HIV serostatus and history of major depressive disorder

HIV and major depressive disorder (MDD) commonly co-occur and are both linked to greater risk-taking behavior, possibly due to neurocognitive impairment (NCI). The present study examined the concordance of the Balloon Analog Risk Task (BART), a gold standard measure of risk-taking propensity, with NCI and real-world sexual risk behaviors in PWH with comorbid MDD. Participants included 259 adults, stratified by HIV serostatus (HIV + /HIV −) and lifetime MDD (MDD + /MDD −), who completed neuropsychological testing, the BART, and sexual risk behavior questionnaires. Logistic regression, stratified by HIV serostatus, examined joint effects of MDD and BART (linear and quadratic) on NCI. Follow-up linear regressions examined sexual risk behavior and neurocognitive domain T-scores as correlates of the BART. NCI prevalence was lowest in HIV − /MDD − , but BART scores did not differ by HIV/MDD status. In the HIV + group, BART performance predicted NCI such that high and low BART scores related to greater odds of NCI, but only in dual-risk HIV + /MDD + individuals. HIV + /MDD + individuals with both low and high BART scores exhibited poorer learning and recall, whereas processing speed and executive function were only poor in low BART risk-taking HIV + /MDD + . Higher BART scores linearly related to higher sexual risk behaviors only in MDD + individuals, independent of HIV serostatus. Low and high risk-taking on the BART may reflect discrete neurocognitive profiles in HIV + /MDD + individuals, with differential implications for real-world sexual risk behavior. HIV and comorbid MDD may disturb corticostriatal circuits responsible for integrating affective and neurocognitive components of decision-making, thereby contributing to risk-averse and risk-taking phenotypes.

Reward/Punishment-Based Decision Making in Rodents

Deficits in decision making are at the heart of many psychiatric diseases, such as substance abuse disorders and attention deficit hyperactivity disorder. Consequently, rodent models of decision making are germane to understanding the neural mechanisms underlying adaptive choice behavior and how such mechanisms can become compromised in pathological conditions. A critical factor that must be integrated with reward value to ensure optimal decision making is the occurrence of consequences, which can differ based on probability (risk of punishment) and temporal contiguity (delayed punishment). This article will focus on two models of decision making that involve explicit punishment, both of which recapitulate different aspects of consequences during human decision making. We will discuss each behavioral protocol, the parameters to consider when designing an experiment, and finally how such animal models can be utilized in studies of psychiatric disease. © 2020 Wiley Periodicals LLC. Basic Protocol 1: Behavioral training Support Protocol: Equipment testing Alternate Protocol: Reward discrimination Basic Protocol 2: Risky decision-making task (RDT) Basic Protocol 3: Delayed punishment decision-making task (DPDT).

Advances in understanding meso-cortico-limbic-striatal systems mediating risky reward seeking

The risk of an aversive consequence occurring as the result of a reward-seeking action can have a profound effect on subsequent behavior. Such aversive events can be described as punishers, as they decrease the probability that the same action will be produced again in the future and increase the exploration of less risky alternatives. Punishment can involve the omission of an expected rewarding event ("negative" punishment) or the addition of an unpleasant event ("positive" punishment). Although many individuals adaptively navigate situations associated with the risk of negative or positive punishment, those suffering from substance use disorders or behavioral addictions tend to be less able to curtail addictive behaviors despite the aversive consequences associated with them. Here, we discuss the psychological processes underpinning reward seeking despite the risk of negative and positive punishment and consider how behavioral assays in animals have been employed to provide insights into the neural mechanisms underlying addictive disorders. We then review the critical contributions of dopamine signaling to punishment learning and risky reward seeking, and address the roles of interconnected ventral striatal, cortical, and amygdala regions to these processes. We conclude by discussing the ample opportunities for future study to clarify critical gaps in the literature, particularly as related to delineating neural contributions to distinct phases of the risky decision-making process.

Dopamine receptors regulate preference between high-effort and high-risk rewards

RATIONALE

Optimal decision-making necessitates evaluation of multiple rewards that are each offset by distinct costs, such as high effort requirement or high risk of failure. The neurotransmitter dopamine is fundamental toward these cost-benefit analyses, and D1-like and D2-like dopamine receptors differently modulate the reward-discounting effects of both effort and risk. However, measuring the role of dopamine in regulating decision-making between options associated with distinct costs exceeds the scope of traditional rodent economic decision-making paradigms.

OBJECTIVES

We developed the effort vs probability economic conflict task (EvP) to model multimodal economic decision-making in rats. This task measures choice between two rewards of uniform magnitude associated with either a high effort requirement or risk of reward omission. We then tested the modulatory effects of systemic cocaine and D1/D2 blockade or activation on the preference between high-effort and high-risk alternatives.

METHODS

In the EvP, two reinforcers of equal magnitude are associated with either (1) an effort requirement that increases throughout the session (1, 5, 10, and 20 lever presses), or (2) a low probability of reward receipt (25% of probabilistic choices). Critically, the reinforcer for each choice is comparable (one pellet), which eliminates the influence of magnitude discrimination on the decision-making process. After establishing the task, the dopamine transporter blocker cocaine and D1/D2 antagonists and agonists were administered prior to EvP performance.

RESULTS

Preference shifted away from either effortful or probabilistic choice when either option became more costly, and this preference was highly variable between subjects and stable over time. Cocaine, D1 activation, and D2 blockade produced limited, dose-dependent shifts in choice preference contingent on high or low effort conditions. In contrast, D2 activation across multiple doses evoked a robust shift from effortful to risky choice that was evident even when clearly disadvantageous.

CONCLUSIONS

The EvP clearly demonstrates that rats can evaluate distinct effortful or risky costs associated with rewards of comparable magnitude, and shift preference away from either option with increasing cost. This preference is more tightly linked to D2 than D1 receptor manipulation, suggesting D2-like receptors as a possible therapeutic target for maladaptive biases toward risk-taking over effort.

Abnormal negative feedback processing in individuals with autistic traits in the Iowa gambling task: Evidence from behavior and event-related potentials

Value-based decision making plays an important role in social interaction. Previous studies have reported that individuals with autism spectrum disorder (ASD) exhibit deficits in terms of decision making. However, it is still unknown clearly whether individuals with high autistic traits within nonclinical populations employ abnormal neural substrates in value-based decision-making. To explore this issue, we investigated value-based decision making and its neural substrates in individuals with high and low autistic traits within a typically developing population who completed the revised Iowa gambling task (IGT) based on measurements of event-related potentials (ERPs). The IGT net scores were significantly lower in the group with high autistic traits than the group with low autistic traits in the fifth and sixth blocks. The ERP results showed that the feedback-related negativity (FRN) amplitude in individuals with high autistic traits allowed slight discrimination between positive and negative feedback in the low-risk option. The event-related spectral perturbations (ERSPs) and inter-trial coherence (ITC) of the theta-band frequency were also lower in the group with high autistic traits than the group with low autistic traits in the loss low-risk option. The results obtained in this study indicate that individuals with high autistic traits exhibit an unusual negative feedback process and relevant neural substrate. The FRN amplitude and theta-band oscillation may comprise a neural index of abnormal decision-making processes in individuals with high autistic traits. This study of a small sample may be considered an important step toward a more comprehensive understanding of the autism "spectrum" within a nonclinical population based on cognitive neuroscience.

Prediction errors and valence: From single units to multidimensional encoding in the amygdala

The amygdala-one of the primary structures of the limbic system-is comprised of interconnected nuclei situated within the temporal lobe. It has a well-established role in the modulation of negative affective states, as well as in fear processing. However, its vast projections with diverse brain regions-ranging from the cortex to the brainstem-are suggestive of its more complex involvement in affective or motivational aspects of cognitive processing. The amygdala can play an invaluable role in context-dependent associative learning, unsigned prediction error learning, influencing outcome selection, and multidimensional encoding. In this review, we delve into the amygdala's role in associative learning and outcome selection, emphasizing its intrinsic involvement in the appropriate context-dependent modulation of motivated behavior.

Loss of Sensitivity to Rewards by Dopamine Neurons May Underlie Age-Related Increased Probability Discounting

Normative aging is known to affect how decisions are made in risky situations. Although important individual variability exists, on average, aging is accompanied by greater risk aversion. Here the behavioral and neural mechanisms of greater risk aversion were examined in young and old rats trained on an instrumental probability discounting task. Consistent with the literature, old rats showed greater discounting of reward value when the probability of obtaining rewards dropped below 100%. Behaviorally, reward magnitude discrimination was the same between young and old rats, and yet these same rats exhibited reduced sensitivity to positive, but not negative, choice outcomes. The latter behavioral result was congruent with additional findings that the aged ventral tegmental neurons (including dopamine cells) were less responsive to rewards when compared to the same cell types recorded from young animals. In sum, it appears that reduced responses of dopamine neurons to rewards contribute to aging-related changes in risky decisions.