Decoding subjective decisions from orbitofrontal cortex

Neuronal Encoding in Prefrontal Cortex during Hierarchical Reinforcement Learning

Reinforcement learning models have proven highly effective for understanding learning in both artificial and biological systems. However, these models have difficulty in scaling up to the complexity of real-life environments. One solution is to incorporate the hierarchical structure of behavior. In hierarchical reinforcement learning, primitive actions are chunked together into more temporally abstract actions, called "options," that are reinforced by attaining a subgoal. These subgoals are capable of generating pseudoreward prediction errors, which are distinct from reward prediction errors that are associated with the final goal of the behavior. Studies in humans have shown that pseudoreward prediction errors positively correlate with activation of ACC. To determine how pseudoreward prediction errors are encoded at the single neuron level, we trained two animals to perform a primate version of the task used to generate these errors in humans. We recorded the electrical activity of neurons in ACC during performance of this task, as well as neurons in lateral prefrontal cortex and OFC. We found that the firing rate of a small population of neurons encoded pseudoreward prediction errors, and these neurons were restricted to ACC. Our results provide support for the idea that ACC may play an important role in encoding subgoals and pseudoreward prediction errors to support hierarchical reinforcement learning. One caveat is that neurons encoding pseudoreward prediction errors were relatively few in number, especially in comparison to neurons that encoded information about the main goal of the task.

Optimal Readout of Correlated Neural Activity in a Decision-Making Circuit

The neural correlates of decision making have been extensively studied with tasks involving a choice between two alternatives that is guided by visual cues. While a large body of work argues for a role of the lateral intraparietal (LIP) region of cortex in these tasks, this role may be confounded by the interaction between LIP and other regions, including medial temporal (MT) cortex. Here, we describe a simplified linear model of decision making that is adapted to two tasks: a motion discrimination and a categorization task. We show that the distinct contribution of MT and LIP may indeed be confounded in these tasks. In particular, we argue that the motion discrimination task relies on a straightforward visuomotor mapping, which leads to redundant information between MT and LIP. The categorization task requires a more complex mapping between visual information and decision behavior, and therefore does not lead to redundancy between MT and LIP. Going further, the model predicts that noise correlations within LIP should be greater in the categorization compared to the motion discrimination task due to the presence of shared inputs from MT. The impact of these correlations on task performance is examined by analytically deriving error estimates of an optimal linear readout for shared and unique inputs. Taken together, results clarify the contribution of MT and LIP to decision making and help characterize the role of noise correlations in these regions.

Robust mixture modeling reveals category-free selectivity in reward region neuronal ensembles

Classification of neurons into clusters based on their response properties is an important tool for gaining insight into neural computations. However, it remains unclear to what extent neurons fall naturally into discrete functional categories. We developed a Bayesian method that models the tuning properties of neural populations as a mixture of multiple types of task-relevant response patterns. We applied this method to data from several cortical and striatal regions in economic choice tasks. In all cases, neurons fell into only two clusters: one multiple-selectivity cluster containing all cells driven by task variables of interest and another of no selectivity for those variables. The single cluster of task-sensitive cells argues against robust categorical tuning in these areas. The no-selectivity cluster was unanticipated and raises important questions about what distinguishes these neurons and what role they play. Moreover, the ability to formally identify these nonselective cells allows for more accurate measurement of ensemble effects by excluding or appropriately down-weighting them in analysis. Our findings provide a valuable tool for analysis of neural data, challenge simple categorization schemes previously proposed for these regions, and place useful constraints on neurocomputational models of economic choice and control. NEW & NOTEWORTHY We present a Bayesian method for formally detecting whether a population of neurons can be naturally classified into clusters based on their response tuning properties. We then examine several data sets of reward system neurons for variables and find in all cases that neurons can be classified into only two categories: a functional class and a non-task-driven class. These results provide important constraints for neural models of the reward system.

Linking dynamic patterns of neural activity in orbitofrontal cortex with decision making

Humans and animals demonstrate extraordinary flexibility in choice behavior, particularly when deciding based on subjective preferences. We evaluate options on different scales, deliberate, and often change our minds. Little is known about the neural mechanisms that underlie these dynamic aspects of decision-making, although neural activity in orbitofrontal cortex (OFC) likely plays a central role. Recent evidence from studies in macaques shows that attention modulates value responses in OFC, and that ensembles of OFC neurons dynamically signal different options during choices. When contexts change, these ensembles flexibly remap to encode the new task. Determining how these dynamic patterns emerge and relate to choices will inform models of decision-making and OFC function.

Using a Simple Neural Network to Delineate Some Principles of Distributed Economic Choice

The brain uses a mixture of distributed and modular organization to perform computations and generate appropriate actions. While the principles under which the brain might perform computations using modular systems have been more amenable to modeling, the principles by which the brain might make choices using distributed principles have not been explored. Our goal in this perspective is to delineate some of those distributed principles using a neural network method and use its results as a lens through which to reconsider some previously published neurophysiological data. To allow for direct comparison with our own data, we trained the neural network to perform binary risky choices. We find that value correlates are ubiquitous and are always accompanied by non-value information, including spatial information (i.e., no pure value signals). Evaluation, comparison, and selection were not distinct processes; indeed, value signals even in the earliest stages contributed directly, albeit weakly, to action selection. There was no place, other than at the level of action selection, at which dimensions were fully integrated. No units were specialized for specific offers; rather, all units encoded the values of both offers in an anti-correlated format, thus contributing to comparison. Individual network layers corresponded to stages in a continuous rotation from input to output space rather than to functionally distinct modules. While our network is likely to not be a direct reflection of brain processes, we propose that these principles should serve as hypotheses to be tested and evaluated for future studies.

Covert shift of attention modulates the value encoding in the orbitofrontal cortex

During value-based decision making, we often evaluate the value of each option sequentially by shifting our attention, even when the options are presented simultaneously. The orbitofrontal cortex (OFC) has been suggested to encode value during value-based decision making. Yet it is not known how its activity is modulated by attention shifts. We investigated this question by employing a passive viewing task that allowed us to disentangle effects of attention, value, choice and eye movement. We found that the attention modulated OFC activity through a winner-take-all mechanism. When we attracted the monkeys’ attention covertly, the OFC neuronal activity reflected the reward value of the newly attended cue. The shift of attention could be explained by a normalization model. Our results strongly argue for the hypothesis that the OFC neuronal activity represents the value of the attended item. They provide important insights toward understanding the OFC’s role in value-based decision making.

Free choice shapes normalized value signals in medial orbitofrontal cortex

Normalization is a common cortical computation widely observed in sensory perception, but its importance in perception of reward value and decision making remains largely unknown. We examined (1) whether normalized value signals occur in the orbitofrontal cortex (OFC) and (2) whether changes in behavioral task context influence the normalized representation of value. We record medial OFC (mOFC) single neuron activity in awake-behaving monkeys during a reward-guided lottery task. mOFC neurons signal the relative values of options via a divisive normalization function when animals freely choose between alternatives. The normalization model, however, performed poorly in a variant of the task where only one of the two possible choice options yields a reward and the other was certain not to yield a reward (so called: “forced choice”). The existence of such context-specific value normalization may suggest that the mOFC contributes valuation signals critical for economic decision making when meaningful alternative options are available.

Neurons in prefrontal areas including the medial orbitofrontal cortex (mOFC) represent the relative reward value of choices. Here the authors report that mOFC neurons implement divisive normalization to encode the relative values of lottery options only when the decision involves free choice.

A neuronal theory of sequential economic choice

Results of recent studies point towards a new framework for the neural bases of economic choice. The principles of this framework include the idea that evaluation is limited to a single option within the focus of attention and that we accept or reject that option relative to the entire set of alternatives. Rejection leads attention to a new option, although it can later switch back to a previously rejected one. The option to which a neuron's firing rate refers is determined dynamically by attention and not stably by labelled lines. Value is always computed relative to the value of rejection. Comparison results not from explicit competition between discrete populations of neurons, but indirectly, as in a horse race, from the fact that the first option whose value crosses a threshold is selected. Consequently, comparison can occur within a single pool of neurons rather than by competition between two or more neuronal populations. The computations that constitute comparison thus occur at multiple levels, including premotor levels, simultaneously (i.e. the brain uses a distributed consensus), and not in discrete stages. This framework suggests a solution to a set of otherwise unresolved neuronal binding problems that result from the need to link options to values, comparisons to actions, and choices to outcomes.

Inverted activity patterns in ventromedial prefrontal cortex during value-guided decision-making in a less-is-more task

Ventromedial prefrontal cortex has been linked to choice evaluation and decision-making in humans but understanding the role it plays is complicated by the fact that little is known about the corresponding area of the macaque brain. We recorded activity in macaques using functional magnetic resonance imaging during two very different value-guided decision-making tasks. In both cases ventromedial prefrontal cortex activity reflected subjective choice values during decision-making just as in humans but the relationship between the blood oxygen level-dependent signal and both decision-making and choice value was inverted and opposite to the relationship seen in humans. In order to test whether the ventromedial prefrontal cortex activity related to choice values is important for decision-making we conducted an additional lesion experiment; lesions that included the same ventromedial prefrontal cortex region disrupted normal subjective evaluation of choices during decision-making.

Ventromedial prefrontal cortex in humans shows functional magnetic resonance imaging signals related to the subjective values of choices that are taken during decision-making as well as task-negative signals. Here, the authors report that in macaque ventromedial prefrontal cortex both activity patterns are inverted and lesions of this area disrupt subjective choice evaluation.

Lateral Orbitofrontal Inactivation Dissociates Devaluation-Sensitive Behavior and Economic Choice

How do we choose between goods that have different subjective values, like apples and oranges? Neuroeconomics proposes that this is done by reducing complex goods to a single unitary value to allow comparison. This value is computed "on the fly" from the underlying model of the goods space, allowing decisions to meet current needs. This is termed "model-based" behavior to distinguish it from pre-determined, habitual, or "model-free" behavior. The lateral orbitofrontal cortex (OFC) supports model-based behavior in rats and primates, but whether the OFC is necessary for economic choice is less clear. Here we tested this question by optogenetically inactivating the lateral OFC in rats in a classic model-based task and during economic choice. Contrary to predictions, inactivation disrupted model-based behavior without affecting economic choice.

Correlates of decisional dynamics in the dorsal anterior cingulate cortex

We hypothesized that during binary economic choice, decision makers use the first option they attend as a default to which they compare the second. To test this idea, we recorded activity of neurons in the dorsal anterior cingulate cortex (dACC) of macaques choosing between gambles presented asynchronously. We find that ensemble encoding of the value of the first offer includes both choice-dependent and choice-independent aspects, as if reflecting a partial decision. That is, its responses are neither entirely pre- nor post-decisional. In contrast, coding of the value of the second offer is entirely decision dependent (i.e., post-decisional). This result holds even when offer-value encodings are compared within the same time period. Additionally, we see no evidence for 2 pools of neurons linked to the 2 offers; instead, all comparison appears to occur within a single functionally homogenous pool of task-selective neurons. These observations suggest that economic choices reflect a context-dependent evaluation of attended options. Moreover, they raise the possibility that value representations reflect, to some extent, a tentative commitment to a choice.

Orbitofrontal Cortex: A Neural Circuit for Economic Decisions

Economic choice behavior entails the computation and comparison of subjective values. A central contribution of neuroeconomics has been to show that subjective values are represented explicitly at the neuronal level. With this result at hand, the field has increasingly focused on the difficult question of where in the brain and how exactly subjective values are compared to make a decision. Here, we review a broad range of experimental and theoretical results suggesting that good-based decisions are generated in a neural circuit within the orbitofrontal cortex (OFC). The main lines of evidence supporting this proposal include the fact that goal-directed behavior is specifically disrupted by OFC lesions, the fact that different groups of neurons in this area encode the input and the output of the decision process, the fact that activity fluctuations in each of these cell groups correlate with choice variability, and the fact that these groups of neurons are computationally sufficient to generate decisions. Results from other brain regions are consistent with the idea that good-based decisions take place in OFC and indicate that value signals inform a variety of mental functions. We also contrast the present proposal with other leading models for the neural mechanisms of economic decisions. Finally, we indicate open questions and suggest possible directions for future research.

Spatiotemporal dynamics of information encoding revealed in orbitofrontal high-gamma

High-gamma signals mirror the tuning and temporal profiles of neurons near a recording electrode in sensory and motor areas. These frequencies appear to aggregate local neuronal activity, but it is unclear how this relationship affects information encoding in high-gamma activity (HGA) in cortical areas where neurons are heterogeneous in selectivity and temporal responses, and are not functionally clustered. Here we report that populations of neurons and HGA recorded from the orbitofrontal cortex (OFC) encode similar information, although there is little correspondence between signals recorded by the same electrode. HGA appears to aggregate heterogeneous neuron activity, such that the spiking of a single cell corresponds to only small increases in HGA. Interestingly, large-scale spatiotemporal dynamics are revealed in HGA, but less apparent in the population of single neurons. Overall, HGA is closely related to neuron activity in OFC, and provides a unique means of studying large-scale spatiotemporal dynamics of information processing.

High gamma activity (HGA) and local neurons encode similar information, but it’s unclear if this is true when neurons are heterogeneous, as in the orbitofrontal cortex (OFC). Here, Rich & Wallis show that HGA in OFC is closely related to neuron firing, but reveals clearer spatiotemporal dynamics.

Elucidating the underlying components of food valuation in the human orbitofrontal cortex

The valuation of food is a fundamental component of our decision-making. Yet little is known about how value signals for food and other rewards are constructed by the brain. Using a food-based decision task in human participants, we found that subjective values can be predicted from beliefs about constituent nutritive attributes of food: protein, fat, carbohydrates and vitamin content. Multivariate analyses of functional MRI data demonstrated that, while food value is represented in patterns of neural activity in both medial and lateral parts of the orbitofrontal cortex (OFC), only the lateral OFC represents the elemental nutritive attributes. Effective connectivity analyses further indicate that information about the nutritive attributes represented in the lateral OFC is integrated within the medial OFC to compute an overall value. These findings provide a mechanistic account for the construction of food value from its constituent nutrients.

The Reduction of Ventrolateral Prefrontal Cortex Gray Matter Volume Correlates with Loss of Economic Rationality in Aging

The population of people above 65 years old continues to grow, and there is mounting evidence that as humans age they are more likely to make errors. However, the specific effect of neuroanatomical aging on the efficiency of economic decision-making is poorly understood. We used whole-brain voxel-based morphometry analysis to determine where reduction of gray matter volume in healthy female and male adults over the age of 65 years correlates with a classic measure of economic irrationality: violations of the Generalized Axiom of Revealed Preference. All participants were functionally normal with Mini-Mental State Examination scores ranging between 26 and 30. While our elders showed the previously reported decline in rationality compared with younger subjects, chronological age per se did not correlate with rationality measures within our population of elders. Instead, reduction of gray matter density in ventrolateral prefrontal cortex correlates tightly with irrational behavior. Interestingly, using a large fMRI sample and meta-analytic tool with Neurosynth, we found that this brain area shows strong coactivation patterns with nearly all of the value-associated regions identified in previous studies. These findings point toward a neuroanatomic locus for economic rationality in the aging brain and highlight the importance of understanding both anatomy and function in the study of aging, cognition, and decision-making.SIGNIFICANCE STATEMENT Age is a crucial factor in decision-making, with older individuals making more errors in choices. Using whole-brain voxel-based morphometry analysis, we found that reduction of gray matter density in ventrolateral prefrontal cortex correlates with economic irrationality: reduced gray matter volume in this area correlates with the frequency and severity of violations of the Generalized Axiom of Revealed Preference. Furthermore, this brain area strongly coactivates with other reward-associated regions identified with Neurosynth. These findings point toward a role for neuroscientific discoveries in shaping long-standing economic views of decision-making.

Basolateral Amygdala to Orbitofrontal Cortex Projections Enable Cue-Triggered Reward Expectations

To make an appropriate decision, one must anticipate potential future rewarding events, even when they are not readily observable. These expectations are generated by using observable information (e.g., stimuli or available actions) to retrieve often quite detailed memories of available rewards. The basolateral amygdala (BLA) and orbitofrontal cortex (OFC) are two reciprocally connected key nodes in the circuitry supporting such outcome-guided behaviors. But there is much unknown about the contribution of this circuit to decision making, and almost nothing known about the whether any contribution is via direct, monosynaptic projections, or the direction of information transfer. Therefore, here we used designer receptor-mediated inactivation of OFC→BLA or BLA→OFC projections to evaluate their respective contributions to outcome-guided behaviors in rats. Inactivation of BLA terminals in the OFC, but not OFC terminals in the BLA, disrupted the selective motivating influence of cue-triggered reward representations over reward-seeking decisions as assayed by Pavlovian-to-instrumental transfer. BLA→OFC projections were also required when a cued reward representation was used to modify Pavlovian conditional goal-approach responses according to the reward's current value. These projections were not necessary when actions were guided by reward expectations generated based on learned action-reward contingencies, or when rewards themselves, rather than stored memories, directed action. These data demonstrate that BLA→OFC projections enable the cue-triggered reward expectations that can motivate the execution of specific action plans and allow adaptive conditional responding.SIGNIFICANCE STATEMENT Deficits anticipating potential future rewarding events are associated with many psychiatric diseases. Presently, we know little about the neural circuits supporting such reward expectation. Here we show that basolateral amygdala to orbitofrontal cortex projections are required for expectations of specific available rewards to influence reward seeking and decision making. The necessity of these projections was limited to situations in which expectations were elicited by reward-predictive cues. These projections therefore facilitate adaptive behavior by enabling the orbitofrontal cortex to use environmental stimuli to generate expectations of potential future rewarding events.

What can neuronal populations tell us about cognition?

Nowadays, it is possible to record the activity of hundreds of cells at the same time in behaving animals. However, these data are often treated and analyzed as if they consisted of many independently recorded neurons. How can neuronal populations be uniquely used to learn about cognition? We describe recent work that shows that populations of simultaneously recorded neurons are fundamental to understand the basis of decision-making, including processes such as ongoing deliberations and decision confidence, which generally fall outside the reach of single-cell analysis. Thus, neuronal population data allow addressing novel questions, but they also come with so far unsolved challenges.

Distinct Roles for the Amygdala and Orbitofrontal Cortex in Representing the Relative Amount of Expected Reward

The same reward can possess different motivational meaning depending upon its magnitude relative to other rewards. To study the neurophysiological mechanisms mediating assignment of motivational meaning, we recorded the activity of neurons in the amygdala and orbitofrontal cortex (OFC) of monkeys during a Pavlovian task in which the relative amount of liquid reward associated with one conditioned stimulus (CS) was manipulated by changing the reward amount associated with a second CS. Anticipatory licking tracked relative reward magnitude, implying that monkeys integrated information about recent rewards to adjust the motivational meaning of a CS. Upon changes in relative reward magnitude, neural responses to reward-predictive cues updated more rapidly in OFC than amygdala, and activity in OFC but not the amygdala was modulated by recent reward history. These results highlight a distinction between the amygdala and OFC in assessing reward history to support the flexible assignment of motivational meaning to sensory cues.

Prefrontal Contribution to Decision-Making under Free-Choice Conditions

Executive function is thought to be the coordinated operation of multiple neural processes and allows to accomplish a current goal flexibly. The most important function of the prefrontal cortex is the executive function. Among a variety of executive functions in which the prefrontal cortex participates, decision-making is one of the most important. Although the prefrontal contribution to decision-making has been examined using a variety of behavioral tasks, recent studies using fMRI have shown that the prefrontal cortex participates in decision-making under free-choice conditions. Since decision-making under free-choice conditions represents the very first stage for any kind of decision-making process, it is important that we understand its neural mechanism. Although few studies have examined this issue while a monkey performed a free-choice task, those studies showed that, when the monkey made a decision to subsequently choose one particular option, prefrontal neurons showing selectivity to that option exhibited transient activation just before presentation of the imperative cue. Further studies have suggested that this transient increase is caused by the irregular fluctuation of spontaneous firing just before cue presentation, which enhances the response to the cue and biases the strength of the neuron's selectivity to the option. In addition, this biasing effect was observed only in neurons that exhibited sustained delay-period activity, indicating that this biasing effect not only influences the animal's decision for an upcoming choice, but also is linked to working memory mechanisms in the prefrontal cortex.

Distinct Hippocampal versus Frontoparietal Network Contributions to Retrieval and Memory-guided Exploration

Memory can profoundly influence new learning, presumably because memory optimizes exploration of to-be-learned material. Although hippocampus and frontoparietal networks have been implicated in memory-guided exploration, their specific and interactive roles have not been identified. We examined eye movements during fMRI scanning to identify neural correlates of the influences of memory retrieval on exploration and learning. After retrieval of one object in a multiobject array, viewing was strategically directed away from the retrieved object toward nonretrieved objects, such that exploration was directed toward to-be-learned content. Retrieved objects later served as optimal reminder cues, indicating that exploration caused memory to become structured around the retrieved content. Hippocampal activity was associated with memory retrieval, whereas frontoparietal activity varied with strategic viewing patterns deployed after retrieval, thus providing spatiotemporal dissociation of memory retrieval from memory-guided learning strategies. Time-lagged fMRI connectivity analyses indicated that hippocampal activity predicted frontoparietal activity to a greater extent for a condition in which retrieval guided exploration occurred than for a passive control condition in which exploration was not influenced by retrieval. This demonstrates network-level interaction effects specific to influences of memory on strategic exploration. These findings show how memory guides behavior during learning and demonstrate distinct yet interactive hippocampal-frontoparietal roles in implementing strategic exploration behaviors that determine the fate of evolving memory representations.

Lateral orbitofrontal cortex anticipates choices and integrates prior with current information

Adaptive behavior requires integrating prior with current information to anticipate upcoming events. Brain structures related to this computation should bring relevant signals from the recent past into the present. Here we report that rats can integrate the most recent prior information with sensory information, thereby improving behavior on a perceptual decision-making task with outcome-dependent past trial history. We find that anticipatory signals in the orbitofrontal cortex about upcoming choice increase over time and are even present before stimulus onset. These neuronal signals also represent the stimulus and relevant second-order combinations of past state variables. The encoding of choice, stimulus and second-order past state variables resides, up to movement onset, in overlapping populations. The neuronal representation of choice before stimulus onset and its build-up once the stimulus is presented suggest that orbitofrontal cortex plays a role in transforming immediate prior and stimulus information into choices using a compact state-space representation.

The orbitofrontal cortex encodes outcomes, expected rewards and values, but it is unclear how this region uses this information to inform action selection. Here, the authors show that lateral orbitofrontal cortex anticipates upcoming choices and combines recent prior information with current sensory information.

Optimizing for generalization in the decoding of internally generated activity in the hippocampus

The decoding of a sensory or motor variable from neural activity benefits from a known ground truth against which decoding performance can be compared. In contrast, the decoding of covert, cognitive neural activity, such as occurs in memory recall or planning, typically cannot be compared to a known ground truth. As a result, it is unclear how decoders of such internally generated activity should be configured in practice. We suggest that if the true code for covert activity is unknown, decoders should be optimized for generalization performance using cross-validation. Using ensemble recording data from hippocampal place cells, we show that this cross-validation approach results in different decoding error, different optimal decoding parameters, and different distributions of error across the decoded variable space. In addition, we show that a minor modification to the commonly used Bayesian decoding procedure, which enables the use of spike density functions, results in substantially lower decoding errors. These results have implications for the interpretation of covert neural activity, and suggest easy-to-implement changes to commonly used procedures across domains, with applications to hippocampal place cells in particular. © 2017 Wiley Periodicals, Inc.

Touch-screen visual reversal learning is mediated by value encoding and signal propagation in the orbitofrontal cortex

Behavioral inflexibility is a common symptom of neuropsychiatric disorders which can have a major detrimental impact on quality of life. While the orbitofrontal cortex (OFC) has been strongly implicated in behavioral flexibility in rodents across paradigms, our understanding of how the OFC mediates these behaviors is rapidly adapting. Here we examined neuronal activity during reversal learning by coupling in vivo electrophysiological recording with a mouse touch-screen learning paradigm to further elucidate the role of the OFC in updating reward value. Single unit and oscillatory activity was recorded during well-learned discrimination and 3 distinct phases of reversal (early, chance and well-learned). During touch-screen performance, OFC neuronal firing tracked rewarded responses following a previous rewarded choice when behavior was well learned, but shifted to primarily track repeated errors following a previous error in early reversal. Spike activity tracked rewarded choices independent of previous trial outcome during chance reversal, and returned to the initial pattern of reward response at criterion. Analysis of spike coupling to oscillatory local field potentials showed that less frequently occurring behaviors had significantly fewer neurons locked to any oscillatory frequency. Together, these data support the role of the OFC in tracking the value of individual choices to inform future responses and suggests that oscillatory signaling may be involved in propagating responses to increase or decrease the likelihood that action is taken in the future. They further support the use of touch-screen paradigms in preclinical studies to more closely model clinical approaches to measuring behavioral flexibility.

A distributed, hierarchical and recurrent framework for reward-based choice

Many accounts of reward-based choice argue for distinct component processes that are serial and functionally localized. In this Opinion article, we argue for an alternative viewpoint, in which choices emerge from repeated computations that are distributed across many brain regions. We emphasize how several features of neuroanatomy may support the implementation of choice, including mutual inhibition in recurrent neural networks and the hierarchical organization of timescales for information processing across the cortex. This account also suggests that certain correlates of value are emergent rather than represented explicitly in the brain.

Identity-Specific Reward Representations in Orbitofrontal Cortex Are Modulated by Selective Devaluation

Goal-directed behavior is sensitive to the current value of expected outcomes. This requires independent representations of specific rewards, which have been linked to orbitofrontal cortex (OFC) function. However, the mechanisms by which the human brain updates specific goals on the fly, and translates those updates into choices, have remained unknown. Here we implemented selective devaluation of appetizing food odors in combination with pattern-based neuroimaging and a decision-making task. We found that in a hungry state, participants chose to smell high-intensity versions of two value-matched food odor rewards. After eating a meal corresponding to one of the two odors, participants switched choices toward the low intensity of the sated odor but continued to choose the high intensity of the nonsated odor. This sensory-specific behavioral effect was mirrored by pattern-based changes in fMRI signal in lateral posterior OFC, where specific reward identity representations were altered after the meal for the sated food odor but retained for the nonsated counterpart. In addition, changes in functional connectivity between the OFC and general value coding in ventromedial prefrontal cortex (vmPFC) predicted individual differences in satiety-related choice behavior. These findings demonstrate how flexible representations of specific rewards in the OFC are updated by devaluation, and how functional connections to vmPFC reflect the current value of outcomes and guide goal-directed behavior.SIGNIFICANCE STATEMENT The orbitofrontal cortex (OFC) is critical for goal-directed behavior. A recent proposal is that OFC fulfills this function by representing a variety of state and task variables ("cognitive maps"), including a conjunction of expected reward identity and value. Here we tested how identity-specific representations of food odor reward are updated by satiety. We found that fMRI pattern-based signatures of reward identity in lateral posterior OFC were modulated after selective devaluation, and that connectivity between this region and general value coding ventromedial prefrontal cortex (vmPFC) predicted choice behavior. These results provide evidence for a mechanism by which devaluation modulates a cognitive map of expected reward in OFC and thereby alters general value signals in vmPFC to guide goal-directed behavior.

Orbitofrontal Cortex Signals Expected Outcomes with Predictive Codes When Stable Contingencies Promote the Integration of Reward History

Memory can inform goal-directed behavior by linking current opportunities to past outcomes. The orbitofrontal cortex (OFC) may guide value-based responses by integrating the history of stimulus-reward associations into expected outcomes, representations of predicted hedonic value and quality. Alternatively, the OFC may rapidly compute flexible "online" reward predictions by associating stimuli with the latest outcome. OFC neurons develop predictive codes when rats learn to associate arbitrary stimuli with outcomes, but the extent to which predictive coding depends on most recent events and the integrated history of rewards is unclear. To investigate how reward history modulates OFC activity, we recorded OFC ensembles as rats performed spatial discriminations that differed only in the number of rewarded trials between goal reversals. The firing rate of single OFC neurons distinguished identical behaviors guided by different goals. When >20 rewarded trials separated goal switches, OFC ensembles developed stable and anticorrelated population vectors that predicted overall choice accuracy and the goal selected in single trials. When <10 rewarded trials separated goal switches, OFC population vectors decorrelated rapidly after each switch, but did not develop anticorrelated firing patterns or predict choice accuracy. The results show that, whereas OFC signals respond rapidly to contingency changes, they predict choices only when reward history is relatively stable, suggesting that consecutive rewarded episodes are needed for OFC computations that integrate reward history into expected outcomes.SIGNIFICANCE STATEMENT Adapting to changing contingencies and making decisions engages the orbitofrontal cortex (OFC). Previous work shows that OFC function can either improve or impair learning depending on reward stability, suggesting that OFC guides behavior optimally when contingencies apply consistently. The mechanisms that link reward history to OFC computations remain obscure. Here, we examined OFC unit activity as rodents performed tasks controlled by contingencies that varied reward history. When contingencies were stable, OFC neurons signaled past, present, and pending events; when contingencies were unstable, past and present coding persisted, but predictive coding diminished. The results suggest that OFC mechanisms require stable contingencies across consecutive episodes to integrate reward history, represent predicted outcomes, and inform goal-directed choices.

Computational principles and models of multisensory integration

Combining information from multiple senses creates robust percepts, speeds up responses, enhances learning, and improves detection, discrimination, and recognition. In this review, I discuss computational models and principles that provide insight into how this process of multisensory integration occurs at the behavioral and neural level. My initial focus is on drift-diffusion and Bayesian models that can predict behavior in multisensory contexts. I then highlight how recent neurophysiological and perturbation experiments provide evidence for a distributed redundant network for multisensory integration. I also emphasize studies which show that task-relevant variables in multisensory contexts are distributed in heterogeneous neural populations. Finally, I describe dimensionality reduction methods and recurrent neural network models that may help decipher heterogeneous neural populations involved in multisensory integration.

Choice Behavior Guided by Learned, But Not Innate, Taste Aversion Recruits the Orbitofrontal Cortex

The ability to select an appropriate behavioral response guided by previous emotional experiences is critical for survival. Although much is known about brain mechanisms underlying emotional associations, little is known about how these associations guide behavior when several choices are available. To address this, we performed local pharmacological inactivations of several cortical regions before retrieval of an aversive memory in choice-based versus no-choice-based conditioned taste aversion (CTA) tasks in rats. Interestingly, we found that inactivation of the orbitofrontal cortex (OFC), but not the dorsal or ventral medial prefrontal cortices, blocked retrieval of choice CTA. However, OFC inactivation left retrieval of no-choice CTA intact, suggesting its role in guiding choice, but not in retrieval of CTA memory. Consistently, OFC activity increased in the choice condition compared with no-choice, as measured with c-Fos immunolabeling. Notably, OFC inactivation did not affect choice behavior when it was guided by innate taste aversion. Consistent with an anterior insular cortex (AIC) involvement in storing taste memories, we found that AIC inactivation impaired retrieval of both choice and no-choice CTA. Therefore, this study provides evidence for OFC's role in guiding choice behavior and shows that this is dissociable from AIC-dependent taste aversion memory. Together, our results suggest that OFC is required and recruited to guide choice selection between options of taste associations relayed from AIC.

SIGNIFICANCE STATEMENT

Survival and mental health depend on being able to choose stimuli not associated with danger. This is particularly important when danger is associated with stimuli that we ingest. Although much is known about the brain mechanisms that underlie associations with dangerous taste stimuli, very little is known about how these stored emotional associations guide behavior when it involves choice. By combining pharmacological and immunohistochemistry tools with taste-guided tasks, our study provides evidence for the key role of orbitofrontal cortex activity in choice behavior and shows that this is dissociable from the adjacent insular cortex-dependent taste aversion memory. Understanding the brain mechanisms that underlie the impact that emotional associations have on survival choice behaviors may lead to better treatments for mental disorders characterized by emotional decision-making deficits.