Complementary contributions of basolateral amygdala and orbitofrontal cortex to value learning under uncertainty

Foraging Under Uncertainty Follows the Marginal Value Theorem with Bayesian Updating of Environment Representations

Foraging theory has been a remarkably successful approach to understanding the behavior of animals in many contexts. In patch-based foraging contexts, the marginal value theorem (MVT) shows that the optimal strategy is to leave a patch when the marginal rate of return declines to the average for the environment. However, the MVT is only valid in deterministic environments whose statistics are known to the forager; naturalistic environments seldom meet these strict requirements. As a result, the strategies used by foragers in naturalistic environments must be empirically investigated. We developed a novel behavioral task and a corresponding computational framework for studying patch-leaving decisions in head-fixed and freely moving mice. We varied between-patch travel time, as well as within-patch reward depletion rate, both deterministically and stochastically. We found that mice adopt patch residence times in a manner consistent with the MVT and not explainable by simple ethologically motivated heuristic strategies. Critically, behavior was best accounted for by a modified form of the MVT wherein environment representations were updated based on local variations in reward timing, captured by a Bayesian estimator and dynamic prior. Thus, we show that mice can strategically attend to, learn from, and exploit task structure on multiple timescales simultaneously, thereby efficiently foraging in volatile environments. The results provide a foundation for applying the systems neuroscience toolkit in freely moving and head-fixed mice to understand the neural basis of foraging under uncertainty.

Dissociable Contributions of Basolateral Amygdala and Ventrolateral Orbitofrontal Cortex to Flexible Learning Under Uncertainty

Reversal learning measures the ability to form flexible associations between choice outcomes with stimuli and actions that precede them. This type of learning is thought to rely on several cortical and subcortical areas, including the highly interconnected orbitofrontal cortex (OFC) and basolateral amygdala (BLA), and is often impaired in various neuropsychiatric and substance use disorders. However, the unique contributions of these regions to stimulus- and action-based reversal learning have not been systematically compared using a chemogenetic approach particularly before and after the first reversal that introduces new uncertainty. Here, we examined the roles of ventrolateral OFC (vlOFC) and BLA during reversal learning. Male and female rats were prepared with inhibitory designer receptors exclusively activated by designer drugs targeting projection neurons in these regions and tested on a series of deterministic and probabilistic reversals during which they learned about stimulus identity or side (left or right) associated with different reward probabilities. Using a counterbalanced within-subject design, we inhibited these regions prior to reversal sessions. We assessed initial and pre-/post-reversal changes in performance to measure learning and adjustments to reversals, respectively. We found that inhibition of the ventrolateral orbitofrontal cortex (vlOFC), but not BLA, eliminated adjustments to stimulus-based reversals. Inhibition of BLA, but not vlOFC, selectively impaired action-based probabilistic reversal learning, leaving deterministic reversal learning intact. vlOFC exhibited a sex-dependent role in early adjustment to action-based reversals, but not in overall learning. These results reveal dissociable roles for BLA and vlOFC in flexible learning and highlight a more crucial role for BLA in learning meaningful changes in the reward environment.

Dissociable contributions of basolateral amygdala and ventrolateral orbitofrontal cortex to flexible learning under uncertainty

Reversal learning measures the ability to form flexible associations between choice outcomes with stimuli and actions that precede them. This type of learning is thought to rely on several cortical and subcortical areas, including highly interconnected orbitofrontal cortex (OFC) and basolateral amygdala (BLA), and is often impaired in various neuropsychiatric and substance use disorders. However, unique contributions of these regions to stimulus- and action-based reversal learning have not been systematically compared using a chemogenetic approach and particularly before and after the first reversal that introduces new uncertainty. Here, we examined the roles of ventrolateral OFC (vlOFC) and BLA during reversal learning. Male and female rats were prepared with inhibitory DREADDs targeting projection neurons in these regions and tested on a series of deterministic and probabilistic reversals during which they learned about stimulus identity or side (left or right) associated with different reward probabilities. Using a counterbalanced within-subject design, we inhibited these regions prior to reversal sessions. We assessed initial and pre-post reversal changes in performance to measure learning and adjustments to reversals, respectively. We found that inhibition of vlOFC, but not BLA, eliminated adjustments to stimulus-based reversals. Inhibition of BLA, but not vlOFC, selectively impaired action-based probabilistic reversal learning, leaving deterministic reversal learning intact. vlOFC exhibited a sex-dependent role in early adjustment to action-based reversals, but not in overall learning. These results reveal dissociable roles for BLA and vlOFC in flexible learning and highlight a more crucial role for BLA in learning meaningful changes in the reward environment.

Organization of orbitofrontal-auditory pathways in the Mongolian gerbil

Sound perception is highly malleable, rapidly adjusting to the acoustic environment and behavioral demands. This flexibility is the result of ongoing changes in auditory cortical activity driven by fluctuations in attention, arousal, or prior expectations. Recent work suggests that the orbitofrontal cortex (OFC) may mediate some of these rapid changes, but the anatomical connections between the OFC and the auditory system are not well characterized. Here, we used virally mediated fluorescent tracers to map the projection from OFC to the auditory midbrain, thalamus, and cortex in a classic animal model for auditory research, the Mongolian gerbil (Meriones unguiculatus). We observed no connectivity between the OFC and the auditory midbrain, and an extremely sparse connection between the dorsolateral OFC and higher order auditory thalamic regions. In contrast, we observed a robust connection between the ventral and medial subdivisions of the OFC and the auditory cortex, with a clear bias for secondary auditory cortical regions. OFC axon terminals were found in all auditory cortical lamina but were significantly more concentrated in the infragranular layers. Tissue-clearing and lightsheet microscopy further revealed that auditory cortical-projecting OFC neurons send extensive axon collaterals throughout the brain, targeting both sensory and non-sensory regions involved in learning, decision-making, and memory. These findings provide a more detailed map of orbitofrontal-auditory connections and shed light on the possible role of the OFC in supporting auditory cognition.

Theta oscillations in anterior cingulate cortex and orbitofrontal cortex differentially modulate accuracy and speed in flexible reward learning

Flexible reward learning relies on frontal cortex, with substantial evidence indicating that anterior cingulate cortex (ACC) and orbitofrontal cortex (OFC) subregions play important roles. Recent studies in both rat and macaque suggest theta oscillations (5-10 Hz) may be a spectral signature that coordinates this learning. However, network-level interactions between ACC and OFC in flexible learning remain unclear. We investigated the learning of stimulus-reward associations using a combination of simultaneous in vivo electrophysiology in dorsal ACC and ventral OFC, partnered with bilateral inhibitory DREADDs in ACC. In freely behaving male and female rats and using a within-subject design, we examined accuracy and speed of response across distinct and precisely defined trial epochs during initial visual discrimination learning and subsequent reversal of stimulus-reward contingencies. Following ACC inhibition, there was a propensity for random responding in early reversal learning, with correct vs. incorrect trials distinguished only from OFC, not ACC, theta power differences in the reversal phase. ACC inhibition also hastened incorrect choices during reversal. This same pattern of change in accuracy and speed was not observed in viral control animals. Thus, characteristics of impaired reversal learning following ACC inhibition are poor deliberation and weak theta signaling of accuracy in this region. The present results also point to OFC theta oscillations as a prominent feature of reversal learning, unperturbed by ACC inhibition.

Basolateral amygdala and orbitofrontal cortex, but not dorsal hippocampus, are necessary for the control of reward-seeking by occasion setters

Reward-seeking in the world is driven by cues that can have ambiguous predictive and motivational value. To produce adaptive, flexible reward-seeking, it is necessary to exploit occasion setters, other distinct features in the environment, to resolve the ambiguity of Pavlovian reward-paired cues. Despite this, very little research has investigated the neurobiological underpinnings of occasion setting, and as a result little is known about which brain regions are critical for occasion setting. To address this, we exploited a recently developed task that was amenable to neurobiological inquiry where a conditioned stimulus is only predictive of reward delivery if preceded in time by the non-overlapping presentation of a separate cue-an occasion setter. This task required male rats to maintain and link cue-triggered expectations across time to produce adaptive reward-seeking. We interrogated the contributions of the basolateral amygdala and orbitofrontal cortex to occasion setting as these regions are thought to be critical for the computation and exploitation of state value, respectively. Reversible inactivation of either structure prior to the occasion-setting task resulted in a profound inability of rats to use the occasion setter to guide reward-seeking. In contrast, inactivation of the dorsal hippocampus, a region fundamental for context-specific responding was without effect nor did inactivation of the basolateral amygdala or orbitofrontal cortex in a standard Pavlovian conditioning preparation affect conditioned responding. We conclude that neural activity within the orbitofrontal cortex and basolateral amygdala circuit is necessary to update and resolve ambiguity in the environment to promote cue-driven reward-seeking.

Role of Orbitofrontal Cortex and Differential Effects of Acute and Chronic Stress on Motor Impulsivity Measured With 1-Choice Serial Reaction Time Test in Male Rats

BACKGROUND

Deficits in motor impulsivity, that is, the inability to inhibit a prepotent response, are frequently observed in psychiatric conditions. Several studies suggest that stress often correlates with higher impulsivity. Among the brain areas affected by stress, the orbitofrontal cortex (OFC) is notable because of its role in impulse control. OFC subregions with unique afferent and efferent circuitry play distinct roles in impulse control, yet it is not clear what OFC subregions are engaged during motor impulsivity tasks.

METHODS

In this study we used a rodent test of motor impulsivity, the 1-choice serial reaction time test, to explore activation of OFC subregions either during a well-learned motor impulsivity task or in a challenge task with a longer wait time that increases premature responding. We also examined the effects of acute inescapable stress, chronic intermittent cold stress and chronic unpredictable stress on motor impulsivity.

RESULTS

Fos expression increased in the lateral OFC and agranular insular cortex during performance in both the mastered and challenge conditions. In the ventral OFC, Fos expression increased only during challenge, and within the medial OFC, Fos was not induced in either condition. Inescapable stress produced a transient effect on premature responses in the mastered task, whereas chronic intermittent cold stress and chronic unpredictable stress altered premature responses in both conditions in ways specific to each stressor.

CONCLUSIONS

These results suggest that different OFC subregions have different roles in motor impulse control, and the effects of stress vary depending on the nature and duration of the stressor.

The orbitofrontal cortex represents advantageous choice in the Iowa gambling task

A good‐based model, the central neurobiological model of economic decision‐making, proposes that the orbitofrontal cortex (OFC) represents binary choice outcome, that is, the chosen good. A good is defined by a group of determinants characterizing the conditions in which the commodity is offered, including commodity type, cost, risk, time delay, and ambiguity. Previous studies have found that the OFC represents the binary choice outcome in decision‐making tasks involving commodity type, cost, risk, and delay. Real‐life decisions are often complex and involve uncertainty, rewards, and penalties; however, whether the OFC represents binary choice outcomes in a complex decision‐making situation, for example, Iowa gambling task (IGT), remains unclear. Here, we propose that the OFC represents binary choice outcome, that is, advantageous choice versus disadvantageous choice, in the IGT. We propose two hypotheses: first, the activity pattern in the human OFC represents an advantageous choice; and second, choice induces an OFC‐related functional network. Using functional magnetic resonance imaging and advanced machine‐learning tools, we found that the OFC represented an advantageous choice in the IGT. The OFC representation of advantageous choice was related to decision‐making performance. Choice modulated the functional connectivity between the OFC and the superior medial gyrus. In conclusion, the OFC represents an advantageous choice during the IGT. In the framework of a good‐based model, the results extend the role of the OFC to complex decision‐making situation when making a binary choice.

Probabilistic Reinforcement Learning and Anhedonia

Despite the prominence of anhedonic symptoms associated with diverse neuropsychiatric conditions, there are currently no approved therapeutics designed to attenuate the loss of responsivity to previously rewarding stimuli. However, the search for improved treatment options for anhedonia has been reinvigorated by a recent reconceptualization of the very construct of anhedonia, including within the Research Domain Criteria (RDoC) initiative. This chapter will focus on the RDoC Positive Valence Systems construct of reward learning generally and sub-construct of probabilistic reinforcement learning specifically. The general framework emphasizes objective measurement of a subject's responsivity to reward via reinforcement learning under asymmetrical probabilistic contingencies as a means to quantify reward learning. Indeed, blunted reward responsiveness and reward learning are central features of anhedonia and have been repeatedly described in major depression. Moreover, these probabilistic reinforcement techniques can also reveal neurobiological mechanisms to aid development of innovative treatment approaches. In this chapter, we describe how investigating reward learning can improve our understanding of anhedonia via the four RDoC-recommended tasks that have been used to probe sensitivity to probabilistic reinforcement contingencies and how such task performance is disrupted in various neuropsychiatric conditions. We also illustrate how reverse translational approaches of probabilistic reinforcement assays in laboratory animals can inform understanding of pharmacological and physiological mechanisms. Next, we briefly summarize the neurobiology of probabilistic reinforcement learning, with a focus on the prefrontal cortex, anterior cingulate cortex, striatum, and amygdala. Finally, we discuss treatment implications and future directions in this burgeoning area.

Foraging with the frontal cortex: A cross-species evaluation of reward-guided behavior

Efficient foraging is essential to survival and depends on frontal cortex in mammals. Because of its role in psychiatric disorders, frontal cortex and its contributions to reward procurement have been studied extensively in both rodents and non-human primates. How frontal cortex of these animal models compares is a source of intense debate. Here we argue that translating findings from rodents to non-human primates requires an appreciation of both the niche in which each animal forages as well as the similarities in frontal cortex anatomy and function. Consequently, we highlight similarities and differences in behavior and anatomy, before focusing on points of convergence in how parts of frontal cortex contribute to distinct aspects of foraging in rats and macaques, more specifically. In doing so, our aim is to emphasize where translation of frontal cortex function between species is clearer, where there is divergence, and where future work should focus. We finish by highlighting aspects of foraging for which have received less attention but we believe are critical to uncovering how frontal cortex promotes survival in each species.

Relative salience signaling within a thalamo-orbitofrontal circuit governs learning rate

Learning to predict rewards is essential for the sustained fitness of animals. Contemporary views suggest that such learning is driven by a reward prediction error (RPE)-the difference between received and predicted rewards. The magnitude of learning induced by an RPE is proportional to the product of the RPE and a learning rate. Here we demonstrate using two-photon calcium imaging and optogenetics in mice that certain functionally distinct subpopulations of ventral/medial orbitofrontal cortex (vmOFC) neurons signal learning rate control. Consistent with learning rate control, trial-by-trial fluctuations in vmOFC activity positively correlate with behavioral updating when the RPE is positive, and negatively correlates with behavioral updating when the RPE is negative. Learning rate is affected by many variables including the salience of a reward. We found that the average reward response of these neurons signals the relative salience of a reward, because it decreases after reward prediction learning or the introduction of another highly salient aversive stimulus. The relative salience signaling in vmOFC is sculpted by medial thalamic inputs. These results support emerging theoretical views that prefrontal cortex encodes and controls learning parameters.

The orbitofrontal cartographer

Theories of orbitofrontal cortex (OFC) function have evolved substantially over the last few decades. There is now a general consensus that the OFC is important for predicting aspects of future events and for using these predictions to guide behavior. Yet the precise content of these predictions and the degree to which OFC contributes to agency contingent upon them has become contentious, with several plausible theories advocating different answers to these questions. In this review we will focus on three of these ideas-the economic value, credit assignment, and cognitive map hypotheses-describing both their successes and failures. We will propose that these failures hint at a more nuanced and perhaps unique role for the OFC, particularly the lateral subdivision, in supporting the proposed functions when an underlying model or map of the causal structures in the environment must be constructed or updated. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The orbitofrontal cortex in temporal cognition

One of the most important factors in decision-making is estimating the value of available options. Subregions of the prefrontal cortex, including the orbitofrontal cortex (OFC), have been deemed essential for this process. Value computations require a complex integration across numerous dimensions, including, reward magnitude, effort, internal state, and time. The importance of the temporal dimension is well illustrated by temporal discounting tasks, in which subjects select between smaller-sooner versus larger-later rewards. The specific role of OFC in telling time and integrating temporal information into decision-making remains unclear. Based on the current literature, in this review we reevaluate current theories of OFC function, accounting for the influence of time. Incorporating temporal information into value estimation and decision-making requires distinct, yet interrelated, forms of temporal information including the ability to tell time, represent time, create temporal expectations, and the ability to use this information for optimal decision-making in a wide range of tasks, including temporal discounting and wagering. We use the term "temporal cognition" to refer to the integrated use of these different aspects of temporal information. We suggest that the OFC may be a critical site for the integration of reward magnitude and delay, and thus important for temporal cognition. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The Role of the Rodent Lateral Orbitofrontal Cortex in Simple Pavlovian Cue-Outcome Learning Depends on Training Experience

The orbitofrontal cortex (OFC) is a critical structure in the flexible control of value-based behaviors. OFC dysfunction is typically only detected when task or environmental contingencies change, against a backdrop of apparently intact initial acquisition and behavior. While intact acquisition following OFC lesions in simple Pavlovian cue-outcome conditioning is often predicted by models of OFC function, this predicted null effect has not been thoroughly investigated. Here, we test the effects of lesions and temporary muscimol inactivation of the rodent lateral OFC on the acquisition of a simple single cue-outcome relationship. Surprisingly, pretraining lesions significantly enhanced acquisition after overtraining, whereas post-training lesions and inactivation significantly impaired acquisition. This impaired acquisition to the cue reflects a disruption of behavioral control and not learning since the cue could also act as an effective blocking stimulus in an associative blocking procedure. These findings suggest that even simple cue-outcome representations acquired in the absence of OFC function are impoverished. Therefore, while OFC function is often associated with flexible behavioral control in complex environments, it is also involved in very simple Pavlovian acquisition where complex cue-outcome relationships are irrelevant to task performance.

Sex-dependent effects of chronic intermittent voluntary alcohol consumption on attentional, not motivational, measures during probabilistic learning and reversal

Background

Forced alcohol (ethanol, EtOH) exposure has been shown to cause significant impairments on reversal learning, a widely-used assay of cognitive flexibility, specifically on fully-predictive, deterministic versions of this task. However, previous studies have not adequately considered voluntary EtOH consumption and sex effects on probabilistic reversal learning. The present study aimed to fill this gap in the literature.

Methods

Male and female Long-Evans rats underwent either 10 weeks of voluntary intermittent 20% EtOH access or water only (H2O) access. Rats were then pretrained to initiate trials and learn stimulus-reward associations via touchscreen response, and subsequently required to select between two visual stimuli, rewarded with probability 0.70 or 0.30. In the final phase, reinforcement contingencies were reversed.

Results

We found significant sex differences on several EtOH-drinking variables, with females reaching a higher maximum EtOH consumption, exhibiting more high-drinking days, and escalating their EtOH at a quicker rate compared to males. During early abstinence, EtOH drinkers (and particularly EtOH-drinking females) made more initiation omissions and were slower to initiate trials than H2O drinking controls, especially during pretraining. A similar pattern in trial initiations was also observed in discrimination, but not in reversal learning. EtOH drinking rats were unaffected in their reward collection and stimulus response times, indicating intact motivation and motor responding. Although there were sex differences in discrimination and reversal phases, performance improved over time. We also observed sex-independent drinking group differences in win-stay and lose-shift strategies specific to the reversal phase.

Conclusions

Females exhibit increased vulnerability to EtOH effects in early learning: there were sex-dependent EtOH effects on attentional measures during pretraining and discrimination phases. We also found sex-independent EtOH effects on exploration strategies during reversal. Future studies should aim to uncover the neural mechanisms for changes in attention and exploration in both acute and prolonged EtOH withdrawal.

Selective impact of lateral orbitofrontal cortex inactivation on reinstatement of alcohol seeking in male Long-Evans rats

The orbitofrontal cortex (OFC) plays a fundamental role in motivated behavior and decision-making. In humans, OFC structure and function is significantly disrupted in drug using and dependent individuals, including those exhibiting chronic alcohol use and alcoholism. In animal models, the OFC has been shown to significantly influence the seeking of non-alcohol drugs of abuse. However direct investigations of the OFC during alcohol seeking and use have been more limited. In the studies reported here, we inactivated lateral (lOFC) or medial OFC (mOFC) subregions in rats during multiple stages of alcohol seeking. After one month of intermittent access to homecage 20% ethanol (EtOH), rats were trained to self-administer EtOH under an FR3 schedule and implanted with cannulae directed to lOFC or mOFC. We inactivated OFC subregions with baclofen/muscimol during EtOH self-administration, extinction, cue-induced reinstatement, and progressive ratio testing to broadly characterize the influence of these subregions on alcohol seeking. There were no significant effects of mOFC or lOFC inactivation during FR3 self-administration, extinction, or progressive ratio self-administration. However, lOFC, and not mOFC, inactivation significantly decreased cue-induced reinstatement of EtOH seeking. These findings contribute new information to the specific impact of OFC manipulation on operant alcohol seeking, support previous studies investigating the role of OFC in seeking and consumption of alcohol and other drugs of abuse, and indicate a specific role for lOFC vs. mOFC in reinstatement.

Adaptive learning under expected and unexpected uncertainty

The outcome of a decision is often uncertain, and outcomes can vary over repeated decisions. Whether decision outcomes should substantially affect behaviour and learning depends on whether they are representative of a typically experienced range of outcomes or signal a change in the reward environment. Successful learning and decision-making therefore require the ability to estimate expected uncertainty (related to the variability of outcomes) and unexpected uncertainty (related to the variability of the environment). Understanding the bases and effects of these two types of uncertainty and the interactions between them - at the computational and the neural level - is crucial for understanding adaptive learning. Here, we examine computational models and experimental findings to distil computational principles and neural mechanisms for adaptive learning under uncertainty.

Lateral orbitofrontal cortex promotes trial-by-trial learning of risky, but not spatial, biases

Individual choices are not made in isolation but are embedded in a series of past experiences, decisions, and outcomes. The effects of past experiences on choices, often called sequential biases, are ubiquitous in perceptual and value-based decision-making, but their neural substrates are unclear. We trained rats to choose between cued guaranteed and probabilistic rewards in a task in which outcomes on each trial were independent. Behavioral variability often reflected sequential effects, including increased willingness to take risks following risky wins, and spatial 'win-stay/lose-shift' biases. Recordings from lateral orbitofrontal cortex (lOFC) revealed encoding of reward history and receipt, and optogenetic inhibition of lOFC eliminated rats' increased preference for risk following risky wins, but spared other sequential effects. Our data show that different sequential biases are neurally dissociable, and the lOFC's role in adaptive behavior promotes learning of more abstract biases (here, biases for the risky option), but not spatial ones.

Contributions of anterior cingulate cortex and basolateral amygdala to decision confidence and learning under uncertainty

The subjective sense of certainty, or confidence, in ambiguous sensory cues can alter the interpretation of reward feedback and facilitate learning. We trained rats to report the orientation of ambiguous visual stimuli according to a spatial stimulus-response rule that must be learned. Following choice, rats could wait a self-timed delay for reward or initiate a new trial. Waiting times increase with discrimination accuracy, demonstrating that this measure can be used as a proxy for confidence. Chemogenetic silencing of BLA shortens waiting times overall whereas ACC inhibition renders waiting times insensitive to confidence-modulating attributes of visual stimuli, suggesting contribution of ACC but not BLA to confidence computations. Subsequent reversal learning is enhanced by confidence. Both ACC and BLA inhibition block this enhancement but via differential adjustments in learning strategies and consistent use of learned rules. Altogether, we demonstrate dissociable roles for ACC and BLA in transmitting confidence and learning under uncertainty.

Primate prefrontal neurons signal economic risk derived from the statistics of recent reward experience

Risk derives from the variation of rewards and governs economic decisions, yet how the brain calculates risk from the frequency of experienced events, rather than from explicit risk-descriptive cues, remains unclear. Here, we investigated whether neurons in dorsolateral prefrontal cortex process risk derived from reward experience. Monkeys performed in a probabilistic choice task in which the statistical variance of experienced rewards evolved continually. During these choices, prefrontal neurons signaled the reward-variance associated with specific objects ('object risk') or actions ('action risk'). Crucially, risk was not derived from explicit, risk-descriptive cues but calculated internally from the variance of recently experienced rewards. Support-vector-machine decoding demonstrated accurate neuronal risk discrimination. Within trials, neuronal signals transitioned from experienced reward to risk (risk updating) and from risk to upcoming choice (choice computation). Thus, prefrontal neurons encode the statistical variance of recently experienced rewards, complying with formal decision variables of object risk and action risk.

Quantity versus quality: Convergent findings in effort-based choice tasks

Organisms must frequently make cost-benefit decisions based on time, risk, and effort in choosing rewards to pursue. Various tasks have been developed to assess effort-based choice in rats, and experimenters have found largely similar results across tasks and brain regions. In this review, we focus primarily on the convergence of different effort-based choice tasks where quality or quantity of reward are manipulated. In the former, the rat is typically presented with the option to work for a preferred reward or select a less preferred, but freely-available reward. In such paradigms, the rewards are of different identities but are confirmed to differ qualitatively in value by a food preference task when both are freely-available. In the latter task type, rats are required to select between higher magnitude versus lower magnitudes of the same reward, but each with a similar effort requirement. We discuss the strengths/limitations of these paradigms, and describe brain regions that have been probed that result in converging or equivocal findings. Results are also reviewed with reference to a need for future work, and the broader impacts and implications of studies probing the mechanisms of effort.

Rodent Models of Adaptive Value Learning and Decision-Making

Real-world decisions are rarely as straightforward as choosing between clearly "good" vs. "bad" options. More often, options must be evaluated carefully because they differ in relative value. For example, we typically learn about (and make decisions between) options in comparison, where one outcome may be more costly or risky than the other. Several neuropsychiatric conditions are characterized by atypical evaluation of effort and risk costs, including major depression, schizophrenia, autism, obsessive-compulsive disorder, and substance use disorders. Aberrant value learning and decision-making have long been considered a cognitive-behavioral endophenotype of these disorders and can be modeled in rodents. This chapter presents two general methodological domains that the experimenter can manipulate in animal decision-making tasks: risk and effort. Here, we present detailed methods of rodent tasks frequently employed within these domains: probabilistic reversal learning (PRL) and effort choice. These tasks recruit regions within rodent frontal cortex, the amygdala, and the striatum, and performance is heavily modulated by dopamine, making these assays highly valid measures in the study of behavioral and substance addictions, in particular.

The role of the orbitofrontal cortex in alcohol use, abuse, and dependence

One of the major functions of the orbitofrontal cortex (OFC) is to promote flexible motivated behavior. It is no surprise, therefore, that recent work has demonstrated a prominent impact of chronic drug use on the OFC and a potential role for OFC disruption in drug abuse and addiction. Among drugs of abuse, the use of alcohol is particularly salient with respect to OFC function. Although a number of studies in humans have implicated OFC dysregulation in alcohol use disorders, animal models investigating the association between OFC and alcohol use are only beginning to be developed, and there is still a great deal to be revealed. The goal of this review is to consider what is currently known regarding the role of the OFC in alcohol use and dependence. I will first provide a brief, general overview of current views of OFC function and its contributions to drug seeking and addiction. I will then discuss research to date related to the OFC and alcohol use, both in human clinical populations and in non-human models. Finally I will consider issues and strategies to guide future study that may identify this brain region as a key player in the transition from moderated to problematic alcohol use and dependence.

Orbital Frontal Cortex Projections to Secondary Motor Cortex Mediate Exploitation of Learned Rules

Animals face the dilemma between exploiting known opportunities and exploring new ones, a decision-making process supported by cortical circuits. While different types of learning may bias exploration, the circumstances and the degree to which bias occurs is unclear. We used an instrumental lever press task in mice to examine whether learned rules generalize to exploratory situations and the cortical circuits involved. We first trained mice to press one lever for food and subsequently assessed how that learning influenced pressing of a second novel lever. Using outcome devaluation procedures we found that novel lever exploration was not dependent on the food value associated with the trained lever. Further, changes in the temporal uncertainty of when a lever press would produce food did not affect exploration. Instead, accrued experience with the instrumental contingency was strongly predictive of test lever pressing with a positive correlation between experience and trained lever exploitation, but not novel lever exploration. Chemogenetic attenuation of orbital frontal cortex (OFC) projection into secondary motor cortex (M2) biased novel lever exploration, suggesting that experience increases OFC-M2 dependent exploitation of learned associations but leaves exploration constant. Our data suggests exploitation and exploration are parallel decision-making systems that do not necessarily compete.

Functional Heterogeneity within Rat Orbitofrontal Cortex in Reward Learning and Decision Making

Rat orbitofrontal cortex (OFC) is located in the dorsal bank of the rhinal sulcus, and is divided into the medial orbital area, ventral orbital area, ventrolateral orbital area, lateral orbital area, dorsolateral orbital area, and agranular insular areas. Over the past 20 years, there has been a marked increase in the number of publications focused on the functions of rat OFC. While collectively this extensive body of work has provided great insight into the functions of OFC, leading to theoretical and computational models of its functions, one issue that has emerged relates to what is defined as OFC because targeting of this region can be quite variable between studies of appetitive behavior, even within the same species. Also apparent is that there is an oversampling and undersampling of certain subregions of rat OFC for study, and this will be demonstrated here. The intent of the Viewpoint is to summarize studies in rat OFC, given the diversity of what groups refer to as "OFC," and to integrate these with the findings of recent anatomical studies. The primary aim is to help discern functions in reward learning and decision-making, clearing the course for future empirical work.