Lateral orbitofrontal cortex promotes trial-by-trial learning of risky, but not spatial, biases

Location- and feature-based selection histories make independent, qualitatively distinct contributions to urgent visuomotor performance

Attention mechanisms that guide visuomotor behaviors are classified into three broad types according to their reliance on stimulus salience, current goals, and selection histories (i.e., recent experience with events of many sorts). These forms of attentional control are clearly distinct and multifaceted, but what is largely unresolved is how they interact dynamically to determine impending visuomotor choices. To investigate this, we trained two macaque monkeys to perform an urgent version of an oddball search task in which a red target appears among three green distracters, or vice versa. By imposing urgency, performance can be tracked continuously as it transitions from uninformed guesses to informed choices, and this, in turn, permits assessment of attentional control as a function of time. We found that the probability of making a correct choice was strongly modulated by the histories of preceding target colors and target locations. Crucially, although both effects were gated by success (or reward), the two variables played dynamically distinct roles: whereas location history promoted an early motor bias, color history modulated the later perceptual evaluation. Furthermore, target color and location influenced performance independently of each other. The results show that, when combined, selection histories can give rise to enormous swings in visuomotor performance even in simple tasks with highly discriminable stimuli.

Tracking subjects' strategies in behavioural choice experiments at trial resolution

Investigating how, when, and what subjects learn during decision-making tasks requires tracking their choice strategies on a trial-by-trial basis. Here, we present a simple but effective probabilistic approach to tracking choice strategies at trial resolution using Bayesian evidence accumulation. We show this approach identifies both successful learning and the exploratory strategies used in decision tasks performed by humans, non-human primates, rats, and synthetic agents. Both when subjects learn and when rules change the exploratory strategies of win-stay and lose-shift, often considered complementary, are consistently used independently. Indeed, we find the use of lose-shift is strong evidence that subjects have latently learnt the salient features of a new rewarded rule. Our approach can be extended to any discrete choice strategy, and its low computational cost is ideally suited for real-time analysis and closed-loop control.

The rat frontal orienting field dynamically encodes value for economic decisions under risk

Frontal and parietal cortex are implicated in economic decision-making, but their causal roles are untested. Here we silenced the frontal orienting field (FOF) and posterior parietal cortex (PPC) while rats chose between a cued lottery and a small stable surebet. PPC inactivations produced minimal short-lived effects. FOF inactivations reliably reduced lottery choices. A mixed-agent model of choice indicated that silencing the FOF caused a change in the curvature of the rats' utility function (U = Vρ). Consistent with this finding, single-neuron and population analyses of neural activity confirmed that the FOF encodes the lottery value on each trial. A dynamical model, which accounts for electrophysiological and silencing results, suggests that the FOF represents the current lottery value to compare against the remembered surebet value. These results demonstrate that the FOF is a critical node in the neural circuit for the dynamic representation of action values for choice under risk.

Orbitofrontal cortex control of striatum leads economic decision-making

Animals must continually evaluate stimuli in their environment to decide which opportunities to pursue, and in many cases these decisions can be understood in fundamentally economic terms. Although several brain regions have been individually implicated in these processes, the brain-wide mechanisms relating these regions in decision-making are unclear. Using an economic decision-making task adapted for rats, we find that neural activity in both of two connected brain regions, the ventrolateral orbitofrontal cortex (OFC) and the dorsomedial striatum (DMS), was required for economic decision-making. Relevant neural activity in both brain regions was strikingly similar, dominated by the spatial features of the decision-making process. However, the neural encoding of choice direction in OFC preceded that of DMS, and this temporal relationship was strongly correlated with choice accuracy. Furthermore, activity specifically in the OFC projection to the DMS was required for appropriate economic decision-making. These results demonstrate that choice information in the OFC is relayed to the DMS to lead accurate economic decision-making.

Increased cortical structural covariance correlates with anhedonia in schizophrenia

Anhedonia is a common symptom in schizophrenia and is closely related to poor functional outcomes. Several lines of evidence reveal that the orbitofrontal cortex plays an important role in anhedonia. In the present study, we aimed to investigate abnormalities in structural covariance within the orbitofrontal subregions, and to further study their role in anticipatory and consummatory anhedonia in schizophrenia. T1 images of 35 schizophrenia patients and 45 healthy controls were obtained. The cortical thickness of 68 cerebral regions parcellated by the Desikan-Killiany (DK) atlas was calculated. The structural covariance within the orbitofrontal subregions was calculated in both schizophrenia and healthy control groups. Stepwise linear regression was performed to examine the relationship between structural covariance and anhedonia in schizophrenia patients. Patients with schizophrenia exhibited higher structural covariance between the left and right medial orbitofrontal thickness, the left lateral orbitofrontal thickness and left pars orbitalis thickness compared to healthy controls (p < 0.05, FDR corrected). This results imply that the increased structural covariance in orbitofrontal thickness may be involved in the process of developing anhedonia in schizophrenia. The result indicated that the increased structural covariance between the left and right medial orbitofrontal thickness might be a protective factor for anticipatory pleasure (B' = 0.420, p = 0.012).

Amygdala-cortical collaboration in reward learning and decision making

Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.

Value representations in the rodent orbitofrontal cortex drive learning, not choice

Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice: the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning: expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here, we employ a recently developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.

Logistic analysis of choice data: A primer

Logistic regressions were developed in economics to model individual choice behavior. In recent years, they have become an important tool in decision neuroscience. Here, I describe and discuss different logistic models, emphasizing the underlying assumptions and possible interpretations. Logistic models may be used to quantify a variety of behavioral traits, including the relative subjective value of different goods, the choice accuracy, risk attitudes, and choice biases. More complex logistic models can be used for choices between good bundles, in cases of nonlinear value functions, and for choices between multiple options. Finally, logistic models can quantify the explanatory power of neuronal activity on choices, thus providing a valid alternative to receiver operating characteristic (ROC) analyses.

Neuronal origins of reduced accuracy and biases in economic choices under sequential offers

Economic choices are characterized by a variety of biases. Understanding their origins is a long-term goal for neuroeconomics, but progress on this front has been limited. Here, we examined choice biases observed when two goods are offered sequentially. In the experiments, rhesus monkeys chose between different juices offered simultaneously or in sequence. Choices under sequential offers were less accurate (higher variability). They were also biased in favor of the second offer (order bias) and in favor of the preferred juice (preference bias). Analysis of neuronal activity recorded in the orbitofrontal cortex revealed that these phenomena emerged at different computational stages. Lower choice accuracy reflected weaker offer value signals (valuation stage), the order bias emerged during value comparison (decision stage), and the preference bias emerged late in the trial (post-comparison). By neuronal measures, each phenomenon reduced the value obtained on average in each trial and was thus costly to the monkey.

Foraging with the frontal cortex: A cross-species evaluation of reward-guided behavior

Efficient foraging is essential to survival and depends on frontal cortex in mammals. Because of its role in psychiatric disorders, frontal cortex and its contributions to reward procurement have been studied extensively in both rodents and non-human primates. How frontal cortex of these animal models compares is a source of intense debate. Here we argue that translating findings from rodents to non-human primates requires an appreciation of both the niche in which each animal forages as well as the similarities in frontal cortex anatomy and function. Consequently, we highlight similarities and differences in behavior and anatomy, before focusing on points of convergence in how parts of frontal cortex contribute to distinct aspects of foraging in rats and macaques, more specifically. In doing so, our aim is to emphasize where translation of frontal cortex function between species is clearer, where there is divergence, and where future work should focus. We finish by highlighting aspects of foraging for which have received less attention but we believe are critical to uncovering how frontal cortex promotes survival in each species.

Relative salience signaling within a thalamo-orbitofrontal circuit governs learning rate

Learning to predict rewards is essential for the sustained fitness of animals. Contemporary views suggest that such learning is driven by a reward prediction error (RPE)-the difference between received and predicted rewards. The magnitude of learning induced by an RPE is proportional to the product of the RPE and a learning rate. Here we demonstrate using two-photon calcium imaging and optogenetics in mice that certain functionally distinct subpopulations of ventral/medial orbitofrontal cortex (vmOFC) neurons signal learning rate control. Consistent with learning rate control, trial-by-trial fluctuations in vmOFC activity positively correlate with behavioral updating when the RPE is positive, and negatively correlates with behavioral updating when the RPE is negative. Learning rate is affected by many variables including the salience of a reward. We found that the average reward response of these neurons signals the relative salience of a reward, because it decreases after reward prediction learning or the introduction of another highly salient aversive stimulus. The relative salience signaling in vmOFC is sculpted by medial thalamic inputs. These results support emerging theoretical views that prefrontal cortex encodes and controls learning parameters.

Subpopulations of neurons in lOFC encode previous and current rewards at time of choice

Studies of neural dynamics in lateral orbitofrontal cortex (lOFC) have shown that subsets of neurons that encode distinct aspects of behavior, such as value, may project to common downstream targets. However, it is unclear whether reward history, which may subserve lOFC's well-documented role in learning, is represented by functional subpopulations in lOFC. Previously, we analyzed neural recordings from rats performing a value-based decision-making task, and we documented trial-by-trial learning that required lOFC (Constantinople et al., 2019). Here, we characterize functional subpopulations of lOFC neurons during behavior, including their encoding of task variables. We found five distinct clusters of lOFC neurons, either based on clustering of their trial-averaged peristimulus time histograms (PSTHs), or a feature space defined by their average conditional firing rates aligned to different task variables. We observed weak encoding of reward attributes, but stronger encoding of reward history, the animal's left or right choice, and reward receipt across all clusters. Only one cluster, however, encoded the animal's reward history at the time shortly preceding the choice, suggesting a possible role in integrating previous and current trial outcomes at the time of choice. This cluster also exhibits qualitatively similar responses to identified corticostriatal projection neurons in a recent study (Hirokawa et al., 2019), and suggests a possible role for subpopulations of lOFC neurons in mediating trial-by-trial learning.

The population doctrine in cognitive neuroscience

A major shift is happening within neurophysiology: a population doctrine is drawing level with the single-neuron doctrine that has long dominated the field. Population-level ideas have so far had their greatest impact in motor neuroscience, but they hold great promise for resolving open questions in cognition as well. Here, we codify the population doctrine and survey recent work that leverages this view to specifically probe cognition. Our discussion is organized around five core concepts that provide a foundation for population-level thinking: (1) state spaces, (2) manifolds, (3) coding dimensions, (4) subspaces, and (5) dynamics. The work we review illustrates the progress and promise that population-level thinking holds for cognitive neuroscience-for delivering new insight into attention, working memory, decision-making, executive function, learning, and reward processing.

Is the core function of orbitofrontal cortex to signal values or make predictions?

One dominant hypothesis about the function of the orbitofrontal cortex (OFC) is that the OFC signals the subjective values of possible outcomes to other brain areas for learning and decision making. This popular view generally neglects the fact that OFC is not necessary for simple value-based behavior (i.e., when values have been directly experienced). An alternative, emerging view suggests that OFC plays a more general role in representing structural information about the task or environment, derived from prior experience, and relevant to predicting behavioral outcomes, such as value. From this perspective, value signaling is simply one derivative of the core underlying function of OFC. New data in favor of both views have been accumulating rapidly. Here we review these new data in discussing the relative merits of these two ideas.

A bidirectional corticoamygdala circuit for the encoding and retrieval of detailed reward memories

Adaptive reward-related decision making often requires accurate and detailed representation of potential available rewards. Environmental reward-predictive stimuli can facilitate these representations, allowing one to infer which specific rewards might be available and choose accordingly. This process relies on encoded relationships between the cues and the sensory-specific details of the rewards they predict. Here, we interrogated the function of the basolateral amygdala (BLA) and its interaction with the lateral orbitofrontal cortex (lOFC) in the ability to learn such stimulus-outcome associations and use these memories to guide decision making. Using optical recording and inhibition approaches, Pavlovian cue-reward conditioning, and the outcome-selective Pavlovian-to-instrumental transfer (PIT) test in male rats, we found that the BLA is robustly activated at the time of stimulus-outcome learning and that this activity is necessary for sensory-specific stimulus-outcome memories to be encoded, so they can subsequently influence reward choices. Direct input from the lOFC was found to support the BLA in this function. Based on prior work, activity in BLA projections back to the lOFC was known to support the use of stimulus-outcome memories to influence decision making. By multiplexing optogenetic and chemogenetic inhibition we performed a serial circuit disconnection and found that the lOFC→BLA and BLA→lOFC pathways form a functional circuit regulating the encoding (lOFC→BLA) and subsequent use (BLA→lOFC) of the stimulus-dependent, sensory-specific reward memories that are critical for adaptive, appetitive decision making.

Advances in understanding meso-cortico-limbic-striatal systems mediating risky reward seeking

The risk of an aversive consequence occurring as the result of a reward-seeking action can have a profound effect on subsequent behavior. Such aversive events can be described as punishers, as they decrease the probability that the same action will be produced again in the future and increase the exploration of less risky alternatives. Punishment can involve the omission of an expected rewarding event ("negative" punishment) or the addition of an unpleasant event ("positive" punishment). Although many individuals adaptively navigate situations associated with the risk of negative or positive punishment, those suffering from substance use disorders or behavioral addictions tend to be less able to curtail addictive behaviors despite the aversive consequences associated with them. Here, we discuss the psychological processes underpinning reward seeking despite the risk of negative and positive punishment and consider how behavioral assays in animals have been employed to provide insights into the neural mechanisms underlying addictive disorders. We then review the critical contributions of dopamine signaling to punishment learning and risky reward seeking, and address the roles of interconnected ventral striatal, cortical, and amygdala regions to these processes. We conclude by discussing the ample opportunities for future study to clarify critical gaps in the literature, particularly as related to delineating neural contributions to distinct phases of the risky decision-making process.

On Koopman mode decomposition and tensor component analysis

Koopman mode decomposition and tensor component analysis [also known as CANDECOMP (canonical decomposition)/PARAFAC (parallel factorization)] are two popular approaches of decomposing high dimensional datasets into modes that capture the most relevant features and/or dynamics. Despite their similar goal, the two methods are largely used by different scientific communities and are formulated in distinct mathematical languages. We examine the two together and show that, under certain conditions on the data, the theoretical decomposition given by the tensor component analysis is the same as that given by Koopman mode decomposition. This provides a "bridge" with which the two communities should be able to more effectively communicate. Our work provides new possibilities for algorithmic approaches to Koopman mode decomposition and tensor component analysis and offers a principled way in which to compare the two methods. Additionally, it builds upon a growing body of work showing that dynamical systems theory and Koopman operator theory, in particular, can be useful for problems that have historically made use of optimization theory.

The orbitofrontal cartographer

Theories of orbitofrontal cortex (OFC) function have evolved substantially over the last few decades. There is now a general consensus that the OFC is important for predicting aspects of future events and for using these predictions to guide behavior. Yet the precise content of these predictions and the degree to which OFC contributes to agency contingent upon them has become contentious, with several plausible theories advocating different answers to these questions. In this review we will focus on three of these ideas-the economic value, credit assignment, and cognitive map hypotheses-describing both their successes and failures. We will propose that these failures hint at a more nuanced and perhaps unique role for the OFC, particularly the lateral subdivision, in supporting the proposed functions when an underlying model or map of the causal structures in the environment must be constructed or updated. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Neuroanatomical and functional substrates of the greed personality trait

Greedy individuals often exhibit more impulsive decision-making and short-sighted behaviors. It has been assumed that altered reward circuitry and prospection network is associated with greed personality trait (GPT). In this study, we first explored the morphological characteristics (i.e., gray matter volume; GMV) of GPT combined with univariate and multivariate pattern analysis (MVPA) approaches. Second, we adopted a revised version of inter-temporal choice task and independently manipulated the amount and delay time of future rewards. Using brain-imaging design, reward- and prospection-related brain activations were assessed and their associations with GPT were further examined. The MVPA results showed that GPT was associated with the GMVs in the right lateral frontal pole cortex, left ventromedial prefrontal cortex, right lateral occipital cortex, and right occipital pole. Additionally, we observed that the amount-relevant brain activations (responding to reward circuitry) in the lateral orbitofrontal cortex were negatively associated with individual's variability in GPT scores, whereas the delay time-relevant brain activations (responding to prospection network system) in the dorsolateral prefrontal cortex, dorsomedial prefrontal cortex, superior parietal lobule, and anterior cingulate cortex were positively associated with individual's variability in GPT scores. These findings not only provide novel insights into the neuroanatomical substrates underlying the human dispositional greed, but also suggest the critical roles of reward and prospection processing on the greed.

Processing in Lateral Orbitofrontal Cortex Is Required to Estimate Subjective Preference during Initial, but Not Established, Economic Choice

The orbitofrontal cortex (OFC) is proposed to be critical to economic decision making. Yet one can inactivate OFC without affecting well-practiced choices. One possible explanation of this lack of effect is that well-practiced decisions are codified into habits or configural-based policies not normally thought to require OFC. Here, we tested this idea by training rats to choose between different pellet pairs across a set of standard offers and then inactivating OFC subregions during choices between novel offers of previously experienced pairs or between novel pairs of previously experienced pellets. Contrary to expectations, controls performed as well on novel as experienced offers yet had difficulty initially estimating their subjective preference on novel pairs, difficulty exacerbated by lateral OFC inactivation. This pattern of results indicates that established economic choice reflects the use of an underlying model or goods space and that lateral OFC is only required for normal behavior when the established framework must incorporate new information.

A reinforcement-learning model of active avoidance behavior: Differences between Sprague Dawley and Wistar-Kyoto rats

Avoidance behavior is a typically adaptive response performed by an organism to avert harmful situations. Individuals differ remarkably in their tendency to acquire and perform new avoidance behaviors, as seen in anxiety disorders where avoidance becomes pervasive and inappropriate. In rodent models of avoidance, the inbred Wistar-Kyoto (WKY) rat demonstrates increased learning and expression of avoidance compared to the outbred Sprague Dawley (SD) rat. However, underlying mechanisms that contribute to these differences are unclear. Computational modeling techniques can help identify factors that may not be easily decipherable from behavioral data alone. Here, we utilize a reinforcement learning (RL) model approach to better understand strain differences in avoidance behavior. An actor-critic model, with separate learning rates for action selection (in the actor) and state evaluation (in the critic), was applied to individual data of avoidance acquisition from a large cohort of WKY and SD rats. Latent parameters were extracted, such as learning rate and subjective reinforcement value of foot shock, that were then compared across groups. The RL model was able to accurately represent WKY and SD avoidance behavior, demonstrating that the model could simulate individual performance. The model determined that the perceived negative value of foot shock was significantly higher in WKY than SD rats, whereas learning rate in the actor was lower in WKY than SD rats. These findings demonstrate the utility of computational modeling in identifying underlying processes that could promote strain differences in behavioral performance.

NMDA Receptors in Visual and Olfactory Sensory Integration in Male Long Evans Rats: A Role for the Orbitofrontal Cortex

Sensory integration (SI) is a cognitive process whereby the brain uses unimodal or multimodal sensory features to create a comprehensive representation of the environment. Integration of sensory input is necessary to achieve a coherent perception of the environment, and to subsequently plan and coordinate action. The neural mechanisms mediating SI are poorly understood; however, recent studies suggest that the regulation of SI involves N-methyl-d-aspartate receptors (NMDARs) in orbitofrontal cortex (OFC). Thus, we tested this hypothesis directly in two experiments using object oddity tests that require SI for visual and olfactory stimuli. First, we blocked NMDARs with acute CPP treatment (i.p., 10 mg/kg) and tested rats in unimodal visual and olfactory SI tests, and respective control unimodal oddity tests that do not require SI. Second, we used intra-OFC infusions of AP5 (30 mM) to examine the role of NMDARs in the OFC in the oddity tests requiring SI. Systemic blockade of NMDARs impaired performance on the visual tests regardless of whether SI was required for determining oddity. In the olfactory tests, systemic treatment with CPP impaired the test requiring SI while sparing olfactory oddity, demonstrating a selective impairment in the olfactory SI. Intra-OFC blockade of NMDARs impaired olfactory SI, without effect on visual SI, demonstrating that intra-OFC NMDARs are essential for olfactory, but not visual SI. The present results are discussed in the context of the function of the OFC and its associated circuitry.

Differential encoding of action selection by orbitofrontal and striatal population dynamics

Survival relies on the ability to flexibly choose between different actions according to varying environmental circumstances. Many lines of evidence indicate that action selection involves signaling in corticostriatal circuits, including the orbitofrontal cortex (OFC) and dorsomedial striatum (DMS). While choice-specific responses have been found in individual neurons from both areas, it is unclear whether populations of OFC or DMS neurons are better at encoding an animal's choice. To address this, we trained head-fixed mice to perform an auditory guided two-alternative choice task, which required moving a joystick forward or backward. We then used silicon microprobes to simultaneously measure the spiking activity of OFC and DMS ensembles, allowing us to directly compare population dynamics between these areas within the same animals. Consistent with previous literature, both areas contained neurons that were selective for specific stimulus-action associations. However, analysis of concurrently recorded ensemble activity revealed that the animal's trial-by-trial behavior could be decoded more accurately from DMS dynamics. These results reveal substantial regional differences in encoding action selection, suggesting that DMS neural dynamics are more specialized than OFC at representing an animal's choice of action.NEW & NOTEWORTHY While previous literature shows that both orbitofrontal cortex (OFC) and dorsomedial striatum (DMS) represent information relevant to selecting specific actions, few studies have directly compared neural signals between these areas. Here we compared OFC and DMS dynamics in mice performing a two-alternative choice task. We found that the animal's choice could be decoded more accurately from DMS population activity. This work provides among the first evidence that OFC and DMS differentially represent information about an animal's selected action.