Multi-level encoding of reward, effort, and choice across the frontal cortex and basal ganglia during cost-benefit decision-making

Adaptive value-guided decision-making requires weighing up the costs and benefits of pursuing an available opportunity. Though neurons across frontal cortical-basal ganglia circuits have been repeatedly shown to represent decision-related parameters, it is unclear whether and how this information is coordinated. To address this question, we performed large-scale single-unit recordings simultaneously across 5 medial/orbital frontal and basal ganglia regions as rats decided whether to pursue varying reward payoffs available at different effort costs. Single neurons encoding combinations of decision variables (reward, effort, and choice) were represented within all recorded regions. Coactive cell assemblies, ensembles of neurons that repeatedly coactivated within short time windows (<25 ms), represented the same decision variables despite the members often having diverse individual coding properties. Together, these findings demonstrate a multi-level encoding structure for cost-benefit computations where individual neurons are coordinated into larger assemblies that can represent task variables independently of their constituent components.

Medial orbitofrontal cortical regulation of different aspects of Pavlovian and instrumental reward seeking

RATIONALE

The medial subregion of the orbitofrontal cortex (mOFC) is thought to play an important role representing the expected outcome of a given course of action, as lesioning or inactivating this cortical region results in the adoption of choice strategies based more on observable (rather than previously learned) information. Despite this, its role in mediating basic associative learning remains to be fully clarified.

OBJECTIVE

The present series of experiments examined the role of the mOFC in (1) Pavlovian conditioned approach, (2) conditioned reinforcement, (3) extinction, and (4) cue-induced reinstatement of food-seeking behavior.

METHODS

Separate cohorts of rats went through Pavlovian or instrumental training. Intra-mOFC infusions of either saline or GABA agonists (to temporarily inactivate neural activity) were given prior to Pavlovian approach, conditioned reinforcement, first or second day of instrumental extinction training, or cue-induced reinstatement test days.

RESULTS

mOFC inactivation increased lever-CS contacts in Pavlovian conditioned approach and (2) had no effect on conditioned reinforcement. These manipulations (3) accelerated within-session instrumental extinction during the initial extinction session, but impaired subsequent extinction learning on drug-free days. (4) mOFC inactivation induced differential effects on reinstatement that depended on baseline performance. mOFC inactivation abolished reinstatement in "Reinstater" rats (who displayed robust responding under control conditions) and robustly increased reinstatement in "Non-Reinstater" rats (who showed little reinstatement under control conditions) suggesting that individual differences in reinstatement may be supported by differences in mOFC mediated representations of expected outcomes.

CONCLUSIONS

These findings have important implications for understanding how the mOFC uses stimulus-outcome and action-outcome expectancies to guide behavior, and how dysfunction within this region may contribute to pathological patterns of reward seeking.

Amygdala-cortical collaboration in reward learning and decision making

Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.

Resting-State Correlations of Fatigue Following Military Deployment

OBJECTIVE

Persistent fatigue is common among military servicemembers returning from deployment, especially those with a history of mild traumatic brain injury (mTBI). The purpose of this study was to characterize fatigue following deployment using the Multidimensional Fatigue Inventory (MFI), a multidimensional self-report instrument. The study was developed to test the hypothesis that if fatigue involves disrupted effort/reward processing, this should manifest as altered basal ganglia functional connectivity as observed in other amotivational states.

METHODS

Twenty-eight current and former servicemembers were recruited and completed the MFI. All 28 participants had a history of at least one mTBI during deployment. Twenty-six participants underwent resting-state functional MRI. To test the hypothesis that fatigue was associated with basal ganglia functional connectivity, the investigators measured correlations between MFI subscale scores and the functional connectivity of the left and right caudate, the putamen, and the globus pallidus with the rest of the brain, adjusting for the presence of depression.

RESULTS

The investigators found a significant correlation between functional connectivity of the left putamen and bilateral superior frontal gyri and mental fatigue scores. No correlations with the other MFI subscales survived multiple comparisons correction.

CONCLUSIONS

This exploratory study suggests that mental fatigue in military servicemembers with a history of deployment with at least one mTBI may be related to increased striatal-prefrontal functional connectivity, independent of depression. A finding of effort/reward mismatch may guide future treatment approaches.

Dissociable and Paradoxical Roles of Rat Medial and Lateral Orbitofrontal Cortex in Visual Serial Reversal Learning

Much evidence suggests that reversal learning is mediated by cortico-striatal circuitries with the orbitofrontal cortex (OFC) playing a prominent role. The OFC is a functionally heterogeneous region, but potential differential roles of lateral (lOFC) and medial (mOFC) portions in visual reversal learning have yet to be determined. We investigated the effects of pharmacological inactivation of mOFC and lOFC on a deterministic serial visual reversal learning task for rats. For reference, we also targeted other areas previously implicated in reversal learning: prelimbic (PrL) and infralimbic (IL) prefrontal cortex, and basolateral amygdala (BLA). Inactivating mOFC and lOFC produced opposite effects; lOFC impairing, and mOFC improving, performance in the early, perseverative phase specifically. Additionally, mOFC inactivation enhanced negative feedback sensitivity, while lOFC inactivation diminished feedback sensitivity in general. mOFC and lOFC inactivation also affected novel visual discrimination learning differently; lOFC inactivation paradoxically improved learning, and mOFC inactivation had no effect. We also observed dissociable roles of the OFC and the IL/PrL. Whereas the OFC inactivation affected only perseveration, IL/PrL inactivation improved learning overall. BLA inactivation did not affect perseveration, but improved the late phase of reversal learning. These results support opponent roles of the rodent mOFC and lOFC in deterministic visual reversal learning.

Medial orbitofrontal cortex dopamine D1/D2 receptors differentially modulate distinct forms of probabilistic decision-making

Efficient decision-making involves weighing the costs and benefits associated with different actions and outcomes to maximize long-term utility. The medial orbitofrontal cortex (mOFC) has been implicated in guiding choice in situations involving reward uncertainty, as inactivation in rats alters choice involving probabilistic rewards. The mOFC receives considerable dopaminergic input, yet how dopamine (DA) modulates mOFC function has been virtually unexplored. Here, we assessed how mOFC D₁ and D₂ receptors modulate two forms of reward seeking mediated by this region, probabilistic reversal learning and probabilistic discounting. Separate groups of well-trained rats received intra-mOFC microinfusions of selective D₁ or D₂ antagonists or agonists prior to task performance. mOFC D₁ and D₂ blockade had opposing effects on performance during probabilistic reversal learning and probabilistic discounting. D₁ blockade impaired, while D₂ blockade increased the number of reversals completed, both mediated by changes in errors and negative feedback sensitivity apparent during the initial discrimination of the task, which suggests changes in probabilistic reinforcement learning rather than flexibility. Similarly, D₁ blockade reduced, while D₂ blockade increased preference for larger/risky rewards. Excess D₁ stimulation had no effect on either task, while excessive D₂ stimulation impaired probabilistic reversal performance, and reduced both profitable risky choice and overall task engagement. These findings highlight a previously uncharacterized role for mOFC DA, showing that D₁ and D₂ receptors play dissociable and opposing roles in different forms of reward-related action selection. Elucidating how DA biases behavior in these situations will expand our understanding of the mechanisms regulating optimal and aberrant decision-making.

Role of the Medial Orbitofrontal Cortex and Ventral Tegmental Area in Effort-Related Responding

The posterior subdivision of the medial orbitofrontal cortex (mOFC-p) mediates the willingness to expend effort to reach a selected goal. However, the neural circuitry through which the mOFC-p modulates effort-related function is as yet unknown. The mOFC-p projects prominently to the posterior ventral tegmental area (pVTA). Therefore, we analyzed the role of the mOFC-p and interactions with the pVTA in effort-related responding using a combination of behavioral, pharmacological, and neural circuit analysis methods in rats. Pharmacological inhibition of the mOFC-p was found to increase lever pressing for food under a progressive ratio (PR) schedule of reinforcement. These findings provide further support for a modulation of effort-related function by the mOFC-p. Then, we investigated effects of disconnecting the mOFC-p and pVTA on PR responding using unilateral pharmacological inhibition of both areas. This asymmetric intervention was also found to increase PR responding suggesting that the mOFC-p controls effort-related function through interactions with the pVTA. Possibly, a reduced excitatory mOFC-p drive on pVTA gamma-aminobutyric acid (GABA)ergic relays disinhibits VTA dopamine neurons which are known to support PR responding. Collectively, our findings suggest that the mOFC-p and pVTA are key components of a neural circuit mediating the willingness to expend effort to reach a goal.

The dopamine depleting agent tetrabenazine alters effort-related decision making as assessed by mouse touchscreen procedures

RATIONALE

Effort-based decision-making tasks allow animals to choose between preferred reinforcers that require high effort to obtain vs. low-effort/low reward options. Mesolimbic dopamine (DA) and related neural systems regulate effort-based choice. Tetrabenazine (TBZ) is a vesicular monoamine transport type-2 inhibitor that blocks DA storage and depletes DA. In humans, TBZ induces motivational dysfunction and depression. TBZ has been shown reliably to induce a low-effort bias in rats, but there are fewer mouse studies.

OBJECTIVES

The present studies used touchscreen operant procedures (Bussey-Saksida chambers) to assess the effects of TBZ on effort-based choice in mice.

METHODS

C57BL6 mice were trained to press an elevated lit panel on the touchscreen on a fixed ratio 1 schedule reinforced by strawberry milkshake, vs. approaching and consuming a concurrently available but less preferred food pellets (Bio-serv).

RESULTS

TBZ (2.0-8.0 mg/kg IP) shifted choice, producing a dose-related decrease in panel pressing but an increase in pellet intake. In contrast, reinforcer devaluation by pre-feeding substantially decreased both panel pressing and pellet intake. In free-feeding choice tests, mice strongly preferred the milkshake vs. the pellets, and TBZ had no effect on milkshake intake or preference, indicating that the TBZ-induced low-effort bias was not due to changes in primary food motivation or preference. TBZ significantly decreased tissue levels of nucleus accumbens DA.

CONCLUSION

The DA depleting agent TBZ induced an effort-related motivational dysfunction in mice, which may have clinical relevance for assessing novel drug targets for their potential use as therapeutic agents in patients with motivation impairments.

Chemogenetic Modulation and Single-Photon Calcium Imaging in Anterior Cingulate Cortex Reveal a Mechanism for Effort-Based Decisions

The ACC is implicated in effort exertion and choices based on effort cost, but it is still unclear how it mediates this cost-benefit evaluation. Here, male rats were trained to exert effort for a high-value reward (sucrose pellets) in a progressive ratio lever-pressing task. Trained rats were then tested in two conditions: a no-choice condition where lever-pressing for sucrose was the only available food option, and a choice condition where a low-value reward (lab chow) was freely available as an alternative to pressing for sucrose. Disruption of ACC, via either chemogenetic inhibition or excitation, reduced lever-pressing in the choice, but not in the no-choice, condition. We next looked for value coding cells in ACC during effortful behavior and reward consumption phases during choice and no-choice conditions. For this, we used in vivo miniaturized fluorescence microscopy to reliably track responses of the same cells and compare how ACC neurons respond during the same effortful behavior where there was a choice versus when there was no-choice. We found that lever-press and sucrose-evoked responses were significantly weaker during choice compared with no-choice sessions, which may have rendered them more susceptible to chemogenetic disruption. Together, findings from our interference experiments and neural recordings suggest that a mechanism by which ACC mediates effortful decisions is in the discrimination of the utility of available options. ACC regulates these choices by providing a stable population code for the relative value of different options.

SIGNIFICANCE STATEMENT The ACC is implicated in effort-based decision-making. Here, we used chemogenetics and in vivo calcium imaging to explore its mechanism. Rats were trained to lever press for a high-value reward and tested in two conditions: a no-choice condition where lever-pressing for the high-value reward was the only option, and a choice condition where a low-value reward was also available. Inhibition or excitation of ACC reduced effort toward the high-value option, but only in the choice condition. Neural responses in ACC were weaker in the choice compared with the no-choice condition. A mechanism by which ACC regulates effortful decisions is in providing a stable population code for the discrimination of the utility of available options.

Involvement of the rodent prelimbic and medial orbitofrontal cortices in goal-directed action: A brief review

Goal-directed action refers to selecting behaviors based on the expectation that they will be reinforced with desirable outcomes. It is typically conceptualized as opposing habit-based behaviors, which are instead supported by stimulus-response associations and insensitive to consequences. The prelimbic prefrontal cortex (PL) is positioned along the medial wall of the rodent prefrontal cortex. It is indispensable for action-outcome-driven (goal-directed) behavior, consolidating action-outcome relationships and linking contextual information with instrumental behavior. In this brief review, we will discuss the growing list of molecular factors involved in PL function. Ventral to the PL is the medial orbitofrontal cortex (mOFC). We will also summarize emerging evidence from rodents (complementing existing literature describing humans) that it too is involved in action-outcome conditioning. We describe experiments using procedures that quantify responding based on reward value, the likelihood of reinforcement, or effort requirements, touching also on experiments assessing food consumption more generally. We synthesize these findings with the argument that the mOFC is essential to goal-directed action when outcome value information is not immediately observable and must be recalled and inferred.

Effects of Chronic Ephedrine Toxicity on Functional Connections, Cell Apoptosis, and CREB-Related Proteins in the Prefrontal Cortex of Rhesus Monkeys

Ephedrine abuse has spread in many parts of the world, severely threatening human health. The mechanism of ephedrine toxicity is still unclear. To explore the possible neural mechanisms of ephedrine toxicity, this study established a non-human primate model of ephedrine exposure, analyzed the functional connectivity changes in its prefrontal cortex through resting state BOLD-fMRI, and then inspected the pathophysiological changes as well as the expression of the cyclic adenosine monophosphate response element-binding protein (CREB), phosphorylated CREB (P-CREB), and CREB target proteins (c-fos and fosB) in the prefrontal cortex. After ephedrine toxicity, we found that the prefrontal cortex of monkeys strengthened its functional connectivity with the brain regions that perform motivation, drive, reward, and learning and memory functions and weakened its functional connectivity with the brain regions that perform cognitive control. These results suggest that ephedrine toxicity causes abnormal neural circuits that lead to the amplification and enhancement of drug-related cues and the weakening and damage of cognitive control function. Histology showed that the neurocytotoxicity of ephedrine can cause neuronal degeneration and apoptosis. Real-time PCR and Western blot showed increased expression of CREB mRNA and CREB/P-CREB/c-fos/fosB protein in the prefrontal cortex after ephedrine toxicity. Collectively, the present study indicates that the enhancement of drug-related cues and the weakening of cognitive control caused by abnormal neural circuits after drug exposure may be a major mechanism of brain function changes caused by ephedrine. These histological and molecular changes may be the pathophysiological basis of brain function changes caused by ephedrine.

Dopamine D1 receptors in the medial orbitofrontal cortex support effort-related responding in rats

Rodent studies on effort-related responding provide a tool to analyze basal aspects of motivation and to model psychiatric motivational dysfunctions reflecting low exertion of effort or reduced behavioral activation. It turned out that dopamine (DA) signaling in brain areas such as nucleus accumbens are essential in regulating effort-related motivational function and could play a major role in motivational dysfunction in psychiatric disorders. Recent rodent studies revealed that the medial orbitofrontal cortex (mOFC) is another key component of the neural circuitry regulating effort-related motivational function. The mOFC receives prominent DA input, however, the behavioral role of mOFC DA signaling is unknown. Here, we investigated whether DA signaling in the mOFC supports effort-related responding in rats. Results demonstrate that an intra-mOFC D1 receptor blockade markedly reduced effort-related responding in a progressive ratio task. Notably, the magnitude of this effect was comparable to the one caused by a systemic DA depletion induced by the VMAT-2 inhibitor tetrabenazine or by a satiety-induced motivational downshift. Collectively, our data show for the first time that D1 receptor activity in the mOFC plays a critical role in high effort responding. These results support findings in humans pointing to a role of the mOFC in effort-related responding. It is well known that the mOFC becomes dysfunctional in depression and schizophrenia. Our data point to the possibility that reduced mOFC DA activity could contribute to effort-related motivational symptoms in these disorders and support the notion that the DA system may be a drug target to treat effort-related motivational symptoms.

Quantity versus quality: Convergent findings in effort-based choice tasks

Organisms must frequently make cost-benefit decisions based on time, risk, and effort in choosing rewards to pursue. Various tasks have been developed to assess effort-based choice in rats, and experimenters have found largely similar results across tasks and brain regions. In this review, we focus primarily on the convergence of different effort-based choice tasks where quality or quantity of reward are manipulated. In the former, the rat is typically presented with the option to work for a preferred reward or select a less preferred, but freely-available reward. In such paradigms, the rewards are of different identities but are confirmed to differ qualitatively in value by a food preference task when both are freely-available. In the latter task type, rats are required to select between higher magnitude versus lower magnitudes of the same reward, but each with a similar effort requirement. We discuss the strengths/limitations of these paradigms, and describe brain regions that have been probed that result in converging or equivocal findings. Results are also reviewed with reference to a need for future work, and the broader impacts and implications of studies probing the mechanisms of effort.

Medial orbitofrontal inactivation does not affect economic choice

How are decisions made between different goods? One theory spanning several fields of neuroscience proposes that their values are distilled to a single common neural currency, the calculation of which allows for rational decisions. The orbitofrontal cortex (OFC) is thought to play a critical role in this process, based on the presence of neural correlates of economic value in lateral OFC in monkeys and medial OFC in humans. We previously inactivated lateral OFC in rats without affecting economic choice behavior. Here we inactivated medial OFC in the same task, again without effect. Behavior in the same rats was disrupted by inactivation during progressive ratio responding previously shown to depend on medial OFC, demonstrating the efficacy of the inactivation. These results indicate that medial OFC is not necessary for economic choice, bolstering the proposal that classic economic choice is likely mediated by multiple, overlapping neural circuits.

Oxygen responses within the nucleus accumbens are associated with individual differences in effort exertion in rats

Goal‐directed motivated behaviour is crucial for everyday life. Such behaviour is often measured, in rodents, under a progressive ratio (PR) schedule of reinforcement. Previous studies have identified a few brain structures critical for supporting PR performance. However, the association between neural activity within these regions and individual differences in effort‐related behaviour is not known. Presently, we used constant potential in vivo oxygen amperometry, a surrogate for functional resonance imaging in rodents, to assess changes in tissue oxygen levels within the nucleus accumbens (NAc) and orbitofrontal cortex (OFC) in male Wistar rats performing a PR task. Within both regions, oxygen responses to rewards increased as the effort exerted to obtain the rewards was larger. Furthermore, higher individual breakpoints were associated with greater magnitude NAc oxygen responses. This association could not be explained by temporal confounds and remained significant when controlling for the different number of completed trials. Animals with higher breakpoints also showed greater magnitude NAc oxygen responses to rewards delivered independently of any behaviour. In contrast, OFC oxygen responses were not associated with individual differences in behavioural performance. The present results suggest that greater NAc oxygen responses following rewards, through a process of incentive motivation, may allow organisms to remain on task for longer and to overcome greater effort costs.

Persistent effect of withdrawal from intravenous methamphetamine self-administration on brain activation and behavioral economic indices involving an effort cost

Exposure to drugs of abuse produces maladaptive changes in cost-benefit decision-making, including the evaluation of time and risk. Studies probing the effects of drug exposure on such evaluations have primarily used experimenter-administered drug regimens. Similarly, while much is known about the neural bases of effort, there have been relatively fewer investigations of the effects of drug experience on effort-based choices. We recently reported that experimenter-administered methamphetamine (meth) resulted in steeper discounting of effort for food rewards in rats, when assessed in protracted withdrawal. Here, we studied rats that underwent withdrawal from weeks of meth intravenous self-administration that later could freely select between a high effort, preferred option (progressive ratio lever pressing for sucrose pellets) versus a low effort, less preferred option (freely-available lab chow). We found decreased effort for the preferred reward and changes in a behavioral economic index demonstrating an increased sensitivity to effort in meth-experienced rats. Critically, the decreased effort for the preferred option was only present in the context of a competing option, not when it was the only option. We also confirmed rats preferred sucrose pellets over chow when both were freely available. These long-lasting changes were accompanied by decreased c-Fos activation in ventral striatum and basolateral amygdala, regions known to be important in effort-based choices. Taken together with our previous observations, these results suggest a robust and enduring effect of meth on value-based decision-making, and point to the underlying neural mechanisms that support the evaluation of an effort cost.