A distributed, hierarchical and recurrent framework for reward-based choice

The macaque anterior cingulate cortex translates counterfactual choice value into actual behavioral change

The neural mechanisms mediating sensory-guided decision making have received considerable attention but animals often pursue behaviors for which there is currently no sensory evidence. Such behaviors are guided by internal representations of choice values that have to be maintained even when these choices are unavailable. We investigated how four macaque monkeys maintained representations of the value of counterfactual choices– choices that could not be taken at the current moment but which could be taken in the future. Using functional magnetic resonance imaging, we found two different patterns of activity co-varying with values of counterfactual choices in a circuit spanning hippocampus, anterior lateral prefrontal cortex, and anterior cingulate cortex (ACC). ACC activity also reflected whether the internal value representations would be translated into actual behavioral change. To establish the causal importance of ACC for this translation process, we used a novel technique, Transcranial Focused Ultrasound Stimulation, to reversibly disrupt ACC activity.

The neural computation of inconsistent choice behavior

Humans are often inconsistent (irrational) when choosing among simple bundles of goods, even without any particular changes to framing or context. However, the neural computations that give rise to such inconsistencies are still unknown. Similar to sensory perception and motor output, we propose that a substantial component of inconsistent behavior is due to variability in the neural computation of value. Here, we develop a novel index that measures the severity of inconsistency of each choice, enabling us to directly trace its neural correlates. We find that the BOLD signal in the vmPFC, ACC, and PCC is correlated with the severity of inconsistency on each trial and with the subjective value of the chosen alternative. This suggests that deviations from rational choice arise in the regions responsible for value computation. We offer a computational model of how variability in value computation is a source of inconsistent choices.

How Reward and Aversion Shape Motivation and Decision Making: A Computational Account

Processing rewarding and aversive signals lies at the core of many adaptive behaviors, including value-based decision making. The brain circuits processing these signals are widespread and include the prefrontal cortex, amygdala and striatum, and their dopaminergic innervation. In this review, we integrate historic findings on the behavioral and neural mechanisms of value-based decision making with recent, groundbreaking work in this area. On the basis of this integrated view, we discuss a neuroeconomic framework of value-based decision making, use this to explain the motivation to pursue rewards and how motivation relates to the costs and benefits associated with different courses of action. As such, we consider substance addiction and overeating as states of altered value-based decision making, in which the expectation of reward chronically outweighs the costs associated with substance use and food consumption, respectively. Together, this review aims to provide a concise and accessible overview of important literature on the neural mechanisms of behavioral adaptation to reward and aversion and how these mediate motivated behaviors.

Dissociable components of the reward circuit are involved in appraisal versus choice

People can evaluate a set of options as a whole, or they can approach those same options with the purpose of making a choice between them. A common network has been implicated across these two types of evaluations, including regions of ventromedial prefrontal cortex and the posterior midline. We test the hypothesis that sub-components of this reward circuit are differentially involved in triggering more automatic appraisal of one’s options (Dorsal Value Network) versus explicitly comparing between those options (Ventral Value Network). Participants undergoing fMRI were instructed to appraise how much they liked a set of products (Like) or to choose the product they most preferred (Choose). Activity in the Dorsal Value Network consistently tracked set liking, across both task-relevant (Like) and task-irrelevant (Choose) trials. In contrast, the Ventral Value Network was particularly sensitive to evaluation condition (more active during Choose than Like trials). Within vmPFC, anatomically distinct regions were dissociated in their sensitivity to choice (ventrally, in medial OFC) versus appraisal (dorsally, in pregenual ACC). Dorsal regions additionally tracked decision certainty across both types of evaluation. These findings suggest that separable mechanisms drive decisions about how good one’s options are versus decisions about which option is best.

Neuronal evidence for good-based economic decisions under variable action costs

Previous work showed that economic decisions can be made independently of spatial contingencies. However, when goods available for choice bear different action costs, the decision necessarily reflects aspects of the action. One possibility is that "stimulus values" are combined with the corresponding action costs in a motor representation, and decisions are then made in actions space. Alternatively, action costs could be integrated with other determinants of value in a non-spatial representation. If so, decisions under variable action costs could take place in goods space. Here, we recorded from orbitofrontal cortex while monkeys chose between different juices offered in variable amounts. We manipulated action costs by varying the saccade amplitude, and we dissociated in time and space offer presentation from action planning. Neurons encoding the binary choice outcome did so well before the presentation of saccade targets, indicating that decisions were made in goods space.

Feature-specific prediction errors and surprise across macaque fronto-striatal circuits

To adjust expectations efficiently, prediction errors need to be associated with the precise features that gave rise to the unexpected outcome, but this credit assignment may be problematic if stimuli differ on multiple dimensions and it is ambiguous which feature dimension caused the outcome. Here, we report a potential solution: neurons in four recorded areas of the anterior fronto-striatal networks encode prediction errors that are specific to feature values of different dimensions of attended multidimensional stimuli. The most ubiquitous prediction error occurred for the reward-relevant dimension. Feature-specific prediction error signals a) emerge on average shortly after non-specific prediction error signals, b) arise earliest in the anterior cingulate cortex and later in dorsolateral prefrontal cortex, caudate and ventral striatum, and c) contribute to feature-based stimulus selection after learning. Thus, a widely-distributed feature-specific eligibility trace may be used to update synaptic weights for improved feature-based attention.

In order to adjust expectations efficiently, prediction errors need to be associated with the features that gave rise to the unexpected outcome. Here, the authors show that neurons in anterior fronto-striatal networks encode prediction errors that are specific to feature values of different stimulus dimensions.

The Neurocognitive Bases of Human Volition

Volition refers to a capacity for endogenous action, particularly goal-directed endogenous action, shared by humans and some other animals. It has long been controversial whether a specific set of cognitive processes for volition exist in the human brain, and much scientific thinking on the topic continues to revolve around traditional metaphysical debates about free will. At its origins, scientific psychology had a strong engagement with volition. This was followed by a period of disenchantment, or even outright hostility, during the second half of the twentieth century. In this review, I aim to reinvigorate the scientific approach to volition by, first, proposing a range of different features that constitute a new, neurocognitively realistic working definition of volition. I then focus on three core features of human volition: its generativity (the capacity to trigger actions), its subjectivity (the conscious experiences associated with initiating voluntary actions), and its teleology (the goal-directed quality of some voluntary actions). I conclude that volition is a neurocognitive process of enormous societal importance and susceptible to scientific investigation.

Activity of Prefrontal Neurons Predict Future Choices during Gambling

Neuronal signals in the prefrontal cortex have been reported to predict upcoming decisions. Such activity patterns are often coupled to perceptual cues indicating correct choices or values of different options. How does the prefrontal cortex signal future decisions when no cues are present but when decisions are made based on internal valuations of past experiences with stochastic outcomes? We trained rats to perform a two-arm bandit-task, successfully adjusting choices between certain-small or possible-big rewards with changing long-term advantages. We discovered specialized prefrontal neurons, whose firing during the encounter of no-reward predicted the subsequent choice of animals, even for unlikely or uncertain decisions and several seconds before choice execution. Optogenetic silencing of the prelimbic cortex exclusively timed to encounters of no reward, provoked animals to excessive gambling for large rewards. Firing of prefrontal neurons during outcome evaluation signals subsequent choices during gambling and is essential for dynamically adjusting decisions based on internal valuations.

Local field potentials in dorsal anterior cingulate sulcus reflect rewards but not travel time costs during foraging

To maximise long-term reward rates, foragers deciding when to leave a patch must compute a decision variable that reflects both the immediately available reward and the time costs associated with travelling to the next patch. Identifying the mechanisms that mediate this computation is central to understanding how brains implement foraging decisions. We previously showed that firing rates of dorsal anterior cingulate sulcus neurons incorporate both variables. This result does not provide information about whether integration of information reflected in dorsal anterior cingulate sulcus spiking activity arises locally or whether it is inherited from upstream structures. Here, we examined local field potentials gathered simultaneously with our earlier recordings. In the majority of recording sites, local field potential spectral bands - specifically theta, beta, and gamma frequency ranges - encoded immediately available rewards but not time costs. The disjunction between information contained in spiking and local field potentials can constrain models of foraging-related processing. In particular, given the proposed link between local field potentials and inputs to a brain area, it raises the possibility that local processing within dorsal anterior cingulate sulcus serves to more fully bind immediate reward and time costs into a single decision variable.

Neural mechanisms for learning self and other ownership

Sense of ownership is a ubiquitous and fundamental aspect of human cognition. Here we used model-based functional magnetic resonance imaging and a novel minimal ownership paradigm to probe the behavioural and neural mechanisms underpinning ownership acquisition for ourselves, friends and strangers. We find a self-ownership bias at multiple levels of behaviour from initial preferences to reaction times and computational learning rates. Ventromedial prefrontal cortex (vmPFC) and anterior cingulate sulcus (ACCs) responded more to self vs. stranger associations, but despite a pervasive neural bias to track self-ownership, no brain area tracked self-ownership exclusively. However, ACC gyrus (ACCg) specifically coded ownership prediction errors for strangers and ownership associative strength for friends and strangers but not for self. Core neural mechanisms for associative learning are biased to learn in reference to self but also engaged when learning in reference to others. In contrast, ACC gyrus exhibits specialization for learning about others.

Value-based attentional capture affects multi-alternative decision making

Humans and other animals often violate economic principles when choosing between multiple alternatives, but the underlying neurocognitive mechanisms remain elusive. A robust finding is that adding a third option can alter the relative preference for the original alternatives, but studies disagree on whether the third option's value decreases or increases accuracy. To shed light on this controversy, we used and extended the paradigm of one study reporting a positive effect. However, our four experiments with 147 human participants and a reanalysis of the original data revealed that the positive effect is neither replicable nor reproducible. In contrast, our behavioral and eye-tracking results are best explained by assuming that the third option's value captures attention and thereby impedes accuracy. We propose a computational model that accounts for the complex interplay of value, attention, and choice. Our theory explains how choice sets and environments influence the neurocognitive processes of multi-alternative decision making.

Hierarchical decision-making produces persistent differences in learning performance

Human organizations are commonly characterized by a hierarchical chain of command that facilitates division of labor and integration of effort. Higher-level employees set the strategic frame that constrains lower-level employees who carry out the detailed operations serving to implement the strategy. Typically, strategy and operational decisions are carried out by different individuals that act over different timescales and rely on different kinds of information. We hypothesize that when such decision processes are hierarchically distributed among different individuals, they produce highly heterogeneous and strongly path-dependent joint learning dynamics. To investigate this, we design laboratory experiments of human dyads facing repeated joint tasks, in which one individual is assigned the role of carrying out strategy decisions and the other operational ones. The experimental behavior generates a puzzling bimodal performance distribution–some pairs learn, some fail to learn after a few periods. We also develop a computational model that mirrors the experimental settings and predicts the heterogeneity of performance by human dyads. Comparison of experimental and simulation data suggests that self-reinforcing dynamics arising from initial choices are sufficient to explain the performance heterogeneity observed experimentally.

Medial orbitofrontal inactivation does not affect economic choice

How are decisions made between different goods? One theory spanning several fields of neuroscience proposes that their values are distilled to a single common neural currency, the calculation of which allows for rational decisions. The orbitofrontal cortex (OFC) is thought to play a critical role in this process, based on the presence of neural correlates of economic value in lateral OFC in monkeys and medial OFC in humans. We previously inactivated lateral OFC in rats without affecting economic choice behavior. Here we inactivated medial OFC in the same task, again without effect. Behavior in the same rats was disrupted by inactivation during progressive ratio responding previously shown to depend on medial OFC, demonstrating the efficacy of the inactivation. These results indicate that medial OFC is not necessary for economic choice, bolstering the proposal that classic economic choice is likely mediated by multiple, overlapping neural circuits.

A Control Theoretic Model of Adaptive Learning in Dynamic Environments

To behave adaptively in environments that are noisy and nonstationary, humans and other animals must monitor feedback from their environment and adjust their predictions and actions accordingly. An understudied approach for modeling these adaptive processes comes from the engineering field of control theory, which provides general principles for regulating dynamical systems, often without requiring a generative model. The proportional-integral-derivative (PID) controller is one of the most popular models of industrial process control. The proportional term is analogous to the "delta rule" in psychology, adjusting estimates in proportion to each error in prediction. The integral and derivative terms augment this update to simultaneously improve accuracy and stability. Here, we tested whether the PID algorithm can describe how people sequentially adjust their predictions in response to new information. Across three experiments, we found that the PID controller was an effective model of participants' decisions in noisy, changing environments. In Experiment 1, we reanalyzed a change-point detection experiment and showed that participants' behavior incorporated elements of PID updating. In Experiments 2-3, we developed a task with gradual transitions that we optimized to detect PID-like adjustments. In both experiments, the PID model offered better descriptions of behavioral adjustments than both the classical delta-rule model and its more sophisticated variant, the Kalman filter. We further examined how participants weighted different PID terms in response to salient environmental events, finding that these control terms were modulated by reward, surprise, and outcome entropy. These experiments provide preliminary evidence that adaptive learning in dynamic environments resembles PID control.

Triple dissociation of attention and decision computations across prefrontal cortex

Naturalistic decision-making typically involves sequential deployment of attention to choice alternatives to gather information before a decision is made. Attention filters how information enters decision circuits, thus implying that attentional control may shape how decision computations unfold. We recorded neuronal activity from three subregions of the prefrontal cortex (PFC) while monkeys performed an attention-guided decision-making task. From the first saccade to decision-relevant information, a triple dissociation of decision- and attention-related computations emerged in parallel across PFC subregions. During subsequent saccades, orbitofrontal cortex activity reflected the value comparison between currently and previously attended information. In contrast, the anterior cingulate cortex carried several signals reflecting belief updating in light of newly attended information, the integration of evidence to a decision bound and an emerging plan for what action to choose. Our findings show how anatomically dissociable PFC representations evolve during attention-guided information search, supporting computations critical for value-guided choice.

Functional brain anatomy of exercise regulation

Knowledge about possible brain mechanisms involved in the regulation of exercise intensity has vastly grown over the last decade. The current review attempts to condense this knowledge currently published with a focus on brain imaging studies. A number of psychological manipulations known to influence exercise intensity are discussed with respect to their possibly underlying brain structures. Although far from forming a complete picture, current knowledge allows to speculate on various possible influences and their corresponding neural bases. Especially, the roles of the insular cortex, anterior cingulate cortex, basal ganglia and prefrontal cortex structures are discussed. Interoceptive signals processed in the insular cortex can influence motor activity likely via anterior cingulate cortex with themselves being influenced by higher order prefrontal cortical regions (e.g., when mediating expectancy effects). Such higher order prefrontal regions can also modulate motivation and thus motor activity by influencing valuation processes in the midbrain and other structures.

Prospection, Perseverance, and Insight in Sequential Behavior

Real-world decisions have benefits occurring only later and dependent on additional decisions taken in the interim. We investigated this in a novel decision-making task in humans (n = 76) while measuring brain activity with fMRI (n = 24). Modeling revealed that participants computed the prospective value of decisions: they planned their future behavior taking into account how their decisions might affect which states they would encounter and how they themselves might respond in these states. They considered their own likely future behavioral biases (e.g., failure to adapt to changes in prospective value) and avoided situations in which they might be prone to such biases. Three neural networks in adjacent medial frontal regions were linked to distinct components of prospective decision making: activity in dorsal anterior cingulate cortex, area 8 m/9, and perigenual anterior cingulate cortex reflected prospective value, anticipated changes in prospective value, and the degree to which prospective value influenced decisions.

Opposing pupil responses to offered and anticipated reward values

Previous studies have shown that the pupils dilate more in anticipation of larger rewards. This finding raises the possibility of a more general association between reward amount and pupil size. We tested this idea by characterizing macaque pupil responses to offered rewards during evaluation and comparison in a binary choice task. To control attention, we made use of a design in which offers occurred in sequence. By looking at pupil responses after choice but before reward, we confirmed the previously observed positive association between pupil size and anticipated reward values. Surprisingly, however, we find that pupil size is negatively correlated with the value of offered gambles before choice, during both evaluation and comparison stages of the task. These results demonstrate a functional distinction between offered and anticipated rewards and present evidence against a narrow version of the simulation hypothesis; the idea that we represent offers by reactivating states associated with anticipating them. They also suggest that pupil size is correlated with relative, not absolute, values of offers, suggestive of an accept-reject model of comparison.

Cognitive computational neuroscience

To learn how cognition is implemented in the brain, we must build computational models that can perform cognitive tasks, and test such models with brain and behavioral experiments. Cognitive science has developed computational models that decompose cognition into functional components. Computational neuroscience has modeled how interacting neurons can implement elementary components of cognition. It is time to assemble the pieces of the puzzle of brain computation and to better integrate these separate disciplines. Modern technologies enable us to measure and manipulate brain activity in unprecedentedly rich ways in animals and humans. However, experiments will yield theoretical insight only when employed to test brain-computational models. Here we review recent work in the intersection of cognitive science, computational neuroscience and artificial intelligence. Computational models that mimic brain information processing during perceptual, cognitive and control tasks are beginning to be developed and tested with brain and behavioral data.

Economic Choice as an Untangling of Options into Actions

We propose that economic choice can be understood as a gradual transformation from a domain of options to one of the actions. We draw an analogy with the idea of untangling information in the form vision system and propose that form vision and economic choice may be two aspects of a larger process that sculpts actions based on sensory inputs. From this viewpoint, choice results from the accumulated effect of repetitions of simple computations. These may consist primarily of relative valuations (evaluations relative to the value of rejection, perhaps in a manner akin to divisive normalization) applied to individual offers. With regard to economic choice, cortical brain regions differ primarily in their position and in what information they prioritize, and do not-with a few exceptions-have categorically distinct roles. Each region's specific contribution is determined largely by its inputs; thus, understanding connectivity is crucial for understanding choice. This view suggests that there is no single site of choice, that there is no meaningful distinction between pre- and post-decisionality, and that there is no explicit representation of value in the brain.

Corticoinsular circuits encode subjective value expectation and violation for effortful goal-directed behavior

The ability to form value estimates is crucial for optimal decision making, especially when not all features of a choice option are known. To date, however, the neural mechanisms for expectation processes under conditions of incomplete information are unknown. Using computational fMRI, we show that ventromedial prefrontal cortex encodes the expected value of a trial. We also observe a distinct network composed of dorsal anterior cingulate, anterior insula, and dorsomedial caudate that encodes an expectation violation or prediction error signal, based on previous trial history. These findings highlight how the brain computes and monitors value-based predictions during effortful goal-directed behavior when choice-relevant information is not fully available.

We are presented with choices each day about how to invest our effort to achieve our goals. Critically, these decisions must frequently be made under conditions of incomplete information, where either the effort required or possible reward to be gained is uncertain. Such choices therefore require the development of potential value estimates to guide effortful goal-directed behavior. To date, however, the neural mechanisms for this expectation process are unknown. Here, we used computational fMRI during an effort-based decision-making task where trial-wise information about effort costs and reward magnitudes was presented separately over time, thereby allowing us to model distinct effort/reward computations as choice-relevant information unfolded. We found that ventromedial prefrontal cortex (vmPFC) encoded expected subjective value. Further, activity in dorsal anterior cingulate (dACC) and anterior insula (aI) reflected both effort discounting as well as a subjective value prediction error signal derived from trial history. While prior studies have identified these regions as being involved in effort-based decision making, these data demonstrate their specific role in the formation and maintenance of subjective value estimates as relevant information becomes available.

Correlates of economic decisions in the dorsal and subgenual anterior cingulate cortices

The anterior cingulate cortex can be divided into distinct ventral (subgenual, sgACC) and dorsal (dACC), portions. The role of dACC in value-based decision-making is hotly debated, while the role of sgACC is poorly understood. We recorded neuronal activity in both regions in rhesus macaques performing a token-gambling task. We find that both encode many of the same variables; including integrated offered values of gambles, primary as well as secondary reward outcomes, number of current tokens and anticipated rewards. Both regions exhibit memory traces for offer values and putative value comparison signals. Both regions use a consistent scheme to encode the value of the attended option. This result suggests that neurones do not appear to be specialized for specific offers (that is, neurones use an attentional as opposed to labelled line coding scheme). We also observed some differences between the two regions: (i) coding strengths in dACC were consistently greater than those in sgACC, (ii) neurones in sgACC responded especially to losses and in anticipation of primary rewards, while those in dACC showed more balanced responding and (iii) responses to the first offer were slightly faster in sgACC. These results indicate that sgACC and dACC have some functional overlap in economic choice, and are consistent with the idea, inspired by neuroanatomy, which sgACC may serve as input to dACC.

Robust mixture modeling reveals category-free selectivity in reward region neuronal ensembles

Classification of neurons into clusters based on their response properties is an important tool for gaining insight into neural computations. However, it remains unclear to what extent neurons fall naturally into discrete functional categories. We developed a Bayesian method that models the tuning properties of neural populations as a mixture of multiple types of task-relevant response patterns. We applied this method to data from several cortical and striatal regions in economic choice tasks. In all cases, neurons fell into only two clusters: one multiple-selectivity cluster containing all cells driven by task variables of interest and another of no selectivity for those variables. The single cluster of task-sensitive cells argues against robust categorical tuning in these areas. The no-selectivity cluster was unanticipated and raises important questions about what distinguishes these neurons and what role they play. Moreover, the ability to formally identify these nonselective cells allows for more accurate measurement of ensemble effects by excluding or appropriately down-weighting them in analysis. Our findings provide a valuable tool for analysis of neural data, challenge simple categorization schemes previously proposed for these regions, and place useful constraints on neurocomputational models of economic choice and control. NEW & NOTEWORTHY We present a Bayesian method for formally detecting whether a population of neurons can be naturally classified into clusters based on their response tuning properties. We then examine several data sets of reward system neurons for variables and find in all cases that neurons can be classified into only two categories: a functional class and a non-task-driven class. These results provide important constraints for neural models of the reward system.

Using a Simple Neural Network to Delineate Some Principles of Distributed Economic Choice

The brain uses a mixture of distributed and modular organization to perform computations and generate appropriate actions. While the principles under which the brain might perform computations using modular systems have been more amenable to modeling, the principles by which the brain might make choices using distributed principles have not been explored. Our goal in this perspective is to delineate some of those distributed principles using a neural network method and use its results as a lens through which to reconsider some previously published neurophysiological data. To allow for direct comparison with our own data, we trained the neural network to perform binary risky choices. We find that value correlates are ubiquitous and are always accompanied by non-value information, including spatial information (i.e., no pure value signals). Evaluation, comparison, and selection were not distinct processes; indeed, value signals even in the earliest stages contributed directly, albeit weakly, to action selection. There was no place, other than at the level of action selection, at which dimensions were fully integrated. No units were specialized for specific offers; rather, all units encoded the values of both offers in an anti-correlated format, thus contributing to comparison. Individual network layers corresponded to stages in a continuous rotation from input to output space rather than to functionally distinct modules. While our network is likely to not be a direct reflection of brain processes, we propose that these principles should serve as hypotheses to be tested and evaluated for future studies.

Robust Encoding of Spatial Information in Orbitofrontal Cortex and Striatum

Knowing whether core reward regions carry information about the positions of relevant objects is crucial for adjudicating between choice models. One limitation of previous studies, including our own, is that spatial positions can be consistently differentially associated with rewards, and thus position can be confounded with attention, motor plans, or target identity. We circumvented these problems by using a task in which value-and thus choices-was determined solely by a frequently changing rule, which was randomized relative to spatial position on each trial. We presented offers asynchronously, which allowed us to control for reward expectation, spatial attention, and motor plans in our analyses. We find robust encoding of the spatial position of both offers and choices in two core reward regions, orbitofrontal Area 13 and ventral striatum, as well as in dorsal striatum of macaques. The trial-by-trial correlation in noise in encoding of position was associated with variation in choice, an effect known as choice probability correlation, suggesting that the spatial encoding is associated with choice and is not incidental to it. Spatial information and reward information are not carried by separate sets of neurons, although the two forms of information are temporally dissociable. These results highlight the ubiquity of multiplexed information in association cortex and argue against the idea that these ostensible reward regions serve as part of a pure value domain.

Neural Mechanisms for Adaptive Learned Avoidance of Mental Effort

Humans tend to avoid mental effort. Previous studies have demonstrated this tendency using various demand-selection tasks; participants generally avoid options associated with higher cognitive demand. However, it remains unclear whether humans avoid mental effort adaptively in uncertain and nonstationary environments. If so, it also remains unclear what neural mechanisms underlie such learned avoidance and whether they remain the same regardless of cognitive-demand types. We addressed these issues by developing novel demand-selection tasks where associations between choice options and cognitive-demand levels change over time, with two variations using mental arithmetic and spatial reasoning problems (males/females: 29:4 and 18:2). Most participants showed avoidance, and their choices depended on the demand experienced on multiple preceding trials. We assumed that participants updated the expected cost of mental effort through experience, and fitted their choices by reinforcement learning models, comparing several possibilities. Model-based fMRI analyses revealed that activity in the dorsomedial and lateral frontal cortices was positively correlated with the trial-by-trial expected cost for the chosen option commonly across the different types of cognitive demand. Analyses also revealed a trend of negative correlation in the ventromedial prefrontal cortex. We further identified correlates of cost-prediction error at time of problem presentation or answering the problem, the latter of which partially overlapped with or were proximal to the correlates of expected cost at time of choice cue in the dorsomedial frontal cortex. These results suggest that humans adaptively learn to avoid mental effort, having neural mechanisms to represent expected cost and cost-prediction error, and the same mechanisms operate for various types of cognitive demand.SIGNIFICANCE STATEMENT In daily life, humans encounter various cognitive demands and tend to avoid high-demand options. However, it remains unclear whether humans avoid mental effort adaptively under dynamically changing environments. If so, it also remains unclear what the underlying neural mechanisms are and whether they operate regardless of cognitive-demand types. To address these issues, we developed novel tasks where participants could learn to avoid high-demand options under uncertain and nonstationary environments. Through model-based fMRI analyses, we found regions whose activity was correlated with the expected mental effort cost, or cost-prediction error, regardless of demand type. These regions overlap, or are adjacent with each other, in the dorsomedial frontal cortex. This finding helps clarify the mechanisms for cognitive-demand avoidance, and provides empirical building blocks for the emerging computational theory of mental effort.

A neuronal theory of sequential economic choice

Results of recent studies point towards a new framework for the neural bases of economic choice. The principles of this framework include the idea that evaluation is limited to a single option within the focus of attention and that we accept or reject that option relative to the entire set of alternatives. Rejection leads attention to a new option, although it can later switch back to a previously rejected one. The option to which a neuron's firing rate refers is determined dynamically by attention and not stably by labelled lines. Value is always computed relative to the value of rejection. Comparison results not from explicit competition between discrete populations of neurons, but indirectly, as in a horse race, from the fact that the first option whose value crosses a threshold is selected. Consequently, comparison can occur within a single pool of neurons rather than by competition between two or more neuronal populations. The computations that constitute comparison thus occur at multiple levels, including premotor levels, simultaneously (i.e. the brain uses a distributed consensus), and not in discrete stages. This framework suggests a solution to a set of otherwise unresolved neuronal binding problems that result from the need to link options to values, comparisons to actions, and choices to outcomes.

Viewing Another Act as You Would Creates Altruistic Desires Towards that Other

There has been growing evidence for the existence of distributed, frequently updating social "indices", which are related to the reputation of others and predict altruism towards them. However, the means by which the brain modifies an index based on experiences is still unknown. This work utilizes recent insights on the role of the anterior cingulate cortex during perspective taking, dorsolateral prefrontal representations of context, the temporoparietal junctions relationship with understanding another's background, and dorsomedial prefrontal activation patterns tracking reputation. It aims to show that cognitive empathy causes comparisons between a target's action and the action one would wish to do in the target's position. It also suggests that viewing a target perform the same action that one would in the target's position creates altruistic desires towards the target. By considering these comparisons as central to understanding prosocial and antisocial motivations, a variety of behavioral studies are better explained. This piece seeks to open questions and discussions on the interplay of those brain regions, suggest future approaches to relationship therapy, and establish fundamentals for multi-agent models aimed at normative sociality.

Simultaneous representation of a spectrum of dynamically changing value estimates during decision making

Decisions are based on value expectations derived from experience. We show that dorsal anterior cingulate cortex and three other brain regions hold multiple representations of choice value based on different timescales of experience organized in terms of systematic gradients across the cortex. Some parts of each area represent value estimates based on recent reward experience while others represent value estimates based on experience over the longer term. The value estimates within these areas interact with one another according to their temporal scaling. Some aspects of the representations change dynamically as the environment changes. The spectrum of value estimates may act as a flexible selection mechanism for combining experience-derived value information with other aspects of value to allow flexible and adaptive decisions in changing environments.

Lateral Orbitofrontal Inactivation Dissociates Devaluation-Sensitive Behavior and Economic Choice

How do we choose between goods that have different subjective values, like apples and oranges? Neuroeconomics proposes that this is done by reducing complex goods to a single unitary value to allow comparison. This value is computed "on the fly" from the underlying model of the goods space, allowing decisions to meet current needs. This is termed "model-based" behavior to distinguish it from pre-determined, habitual, or "model-free" behavior. The lateral orbitofrontal cortex (OFC) supports model-based behavior in rats and primates, but whether the OFC is necessary for economic choice is less clear. Here we tested this question by optogenetically inactivating the lateral OFC in rats in a classic model-based task and during economic choice. Contrary to predictions, inactivation disrupted model-based behavior without affecting economic choice.

Correlates of decisional dynamics in the dorsal anterior cingulate cortex

We hypothesized that during binary economic choice, decision makers use the first option they attend as a default to which they compare the second. To test this idea, we recorded activity of neurons in the dorsal anterior cingulate cortex (dACC) of macaques choosing between gambles presented asynchronously. We find that ensemble encoding of the value of the first offer includes both choice-dependent and choice-independent aspects, as if reflecting a partial decision. That is, its responses are neither entirely pre- nor post-decisional. In contrast, coding of the value of the second offer is entirely decision dependent (i.e., post-decisional). This result holds even when offer-value encodings are compared within the same time period. Additionally, we see no evidence for 2 pools of neurons linked to the 2 offers; instead, all comparison appears to occur within a single functionally homogenous pool of task-selective neurons. These observations suggest that economic choices reflect a context-dependent evaluation of attended options. Moreover, they raise the possibility that value representations reflect, to some extent, a tentative commitment to a choice.

Orbitofrontal Cortex: A Neural Circuit for Economic Decisions

Economic choice behavior entails the computation and comparison of subjective values. A central contribution of neuroeconomics has been to show that subjective values are represented explicitly at the neuronal level. With this result at hand, the field has increasingly focused on the difficult question of where in the brain and how exactly subjective values are compared to make a decision. Here, we review a broad range of experimental and theoretical results suggesting that good-based decisions are generated in a neural circuit within the orbitofrontal cortex (OFC). The main lines of evidence supporting this proposal include the fact that goal-directed behavior is specifically disrupted by OFC lesions, the fact that different groups of neurons in this area encode the input and the output of the decision process, the fact that activity fluctuations in each of these cell groups correlate with choice variability, and the fact that these groups of neurons are computationally sufficient to generate decisions. Results from other brain regions are consistent with the idea that good-based decisions take place in OFC and indicate that value signals inform a variety of mental functions. We also contrast the present proposal with other leading models for the neural mechanisms of economic decisions. Finally, we indicate open questions and suggest possible directions for future research.

Functional Heterogeneity within Rat Orbitofrontal Cortex in Reward Learning and Decision Making

Rat orbitofrontal cortex (OFC) is located in the dorsal bank of the rhinal sulcus, and is divided into the medial orbital area, ventral orbital area, ventrolateral orbital area, lateral orbital area, dorsolateral orbital area, and agranular insular areas. Over the past 20 years, there has been a marked increase in the number of publications focused on the functions of rat OFC. While collectively this extensive body of work has provided great insight into the functions of OFC, leading to theoretical and computational models of its functions, one issue that has emerged relates to what is defined as OFC because targeting of this region can be quite variable between studies of appetitive behavior, even within the same species. Also apparent is that there is an oversampling and undersampling of certain subregions of rat OFC for study, and this will be demonstrated here. The intent of the Viewpoint is to summarize studies in rat OFC, given the diversity of what groups refer to as "OFC," and to integrate these with the findings of recent anatomical studies. The primary aim is to help discern functions in reward learning and decision-making, clearing the course for future empirical work.

Can Neuroscience Assist Us in Constructing Better Patterns of Economic Decision-Making?

We draw on outstanding research (Sanfey et al., 2006; McCabe, 2008; Bernheim, 2009; Camerer, 2013; Radu and McClure, 2013; Declerck and Boone, 2016) to substantiate that neuroeconomics covers the investigation of the biological microfoundations of economic cognition and economic conduct, attempts to prove that a superior grasp of how choices are made brings about superior expectations regarding which options are selected, preserves the strictness of economic analysis in defining value-based decision, and associates imaging techniques with economic pattern to explain how individuals decide on a strategy taking into account various possible choices. Neuroeconomics is adequately prepared to regulate the notion of how choices are determined by mental states. The position that will be elaborated in this article is that neuroeconomic patterns are enabled and enhanced in descriptive capacity by psychological outcomes and substantiated in biological processes. Advancement in neuroeconomics takes place when outcomes from distinct procedures are coherent with an ordinary mechanistic clarification of what generates choice, construed by a computational pattern. We will develop this point further by proving that economics improves the concerted effort of neuroeconomics by using its observations in the various results that may stem from the planned and market interplays of diverse participants, and via a series of accurate, explicit, mathematical patterns to construe such interplays and results. Neuroeconomics experiments employ a mixture of brain imaging/stimulation tests advanced in the cognitive neurosciences and microeconomic systems/game theory tests advanced in the economic sciences. Our analyses indicate that neuroeconomics aims to employ the supplementary input gained from brain investigations, associated with the decision maker's selection, with the purpose of better grasping the cogitation process and to utilize the outcomes to enhance economic patterns.

Contrasting Effects of Medial and Lateral Orbitofrontal Cortex Lesions on Credit Assignment and Decision-Making in Humans

The orbitofrontal cortex is critical for goal-directed behavior. Recent work in macaques has suggested the lateral orbitofrontal cortex (lOFC) is relatively more concerned with assignment of credit for rewards to particular choices during value-guided learning, whereas the medial orbitofrontal cortex (often referred to as ventromedial prefrontal cortex in humans; vmPFC/mOFC) is involved in constraining the decision to the relevant options. We examined whether people with damage restricted to subregions of prefrontal cortex showed the patterns of impairment observed in prior investigations of the effects of lesions to homologous regions in macaques. Groups of patients with either lOFC (predominantly right hemisphere), mOFC/vmPFC, or dorsomedial prefrontal (DMF), and a comparison group of healthy age- and education-matched controls performed a probabilistic 3-choice decision-making task. We report anatomically specific patterns of impairment. We found that credit assignment, as indexed by the normal influence of contingent relationships between choice and reward, is reduced in lOFC patients compared with Controls and mOFC/vmPFC patients. Moreover, the effects of reward contingency on choice were similar for patients with lesions in DMF or mOFC/vmPFC, compared with Controls. By contrast, mOFC/vmPFC-lesioned patients made more stochastic choices than Controls when the decision was framed by valuable distracting alternatives, suggesting that value comparisons were no longer independent of irrelevant options. Once again, there was evidence of regional specialization: patients with lOFC lesions were unimpaired relative to Controls. As in macaques, human lOFC and mOFC/vmPFC are necessary for contingent learning and value-guided decision-making, respectively.SIGNIFICANCE STATEMENT The lateral and medial regions of the orbitofrontal cortex are cytoarchitectonically distinct and have different anatomical connections. Previous investigations in macaques have shown these anatomical differences are accompanied by functional specialization for learning and decision-making. Here, for the first time, we test the predictions made by macaque studies in an experiment with humans with frontal lobe lesions, asking whether behavioral impairments can be linked to lateral or medial orbitofrontal cortex. Using equivalent tasks and computational analyses, our findings broadly replicate the pattern reported after selective lesions in monkeys. Patients with lateral orbitofrontal damage had impaired credit assignment, whereas damage to medial orbitofrontal cortex meant that patients were more likely to be distracted by irrelevant options.

Reactivation of associative structure specific outcome responses during prospective evaluation in reward-based choices

Before making a reward-based choice, we must evaluate each option. Some theories propose that prospective evaluation involves a reactivation of the neural response to the outcome. Others propose that it calls upon a response pattern that is specific to each underlying associative structure. We hypothesize that these views are reconcilable: during prospective evaluation, offers reactivate neural responses to outcomes that are unique to each associative structure; when the outcome occurs, this pattern is activated, simultaneously, with a general response to the reward. We recorded single-units from macaque orbitofrontal cortex (Area 13) in a riskless choice task with interleaved described and experienced offer trials. Here we report that neural activations to offers and their outcomes overlap, as do neural activations to the outcomes on the two trial types. Neural activations to experienced and described offers are unrelated even though they predict the same outcomes. Our reactivation theory parsimoniously explains these results.

How the brain evaluates options to make a reward-based choice is unclear. Here, authors show that, prior to choice, neural activity patterns to the potential outcomes are reactivated in macaque orbitofrontal cortex, in a way that reflects the unique event sequences leading up to the outcomes.

Prefrontal Cortex in Control: Broadening the Scope to Identify Mechanisms

Sometime in the past two decades, neuroimaging and behavioral research converged on pFC as an important locus of cognitive control and decision-making, and that seems to be the last thing anyone has agreed on since. Every year sees an increase in the number of roles and functions attributed to distinct subregions within pFC, roles that may explain behavior and neural activity in one context but might fail to generalize across the many behaviors in which each region is implicated. Emblematic of this ongoing proliferation of functions is dorsal ACC (dACC). Novel tasks that activate dACC are followed by novel interpretations of dACC function, and each new interpretation adds to the number of functionally specific processes contained within the region. This state of affairs, a recurrent and persistent behavior followed by an illusory and transient relief, can be likened to behavioral pathology. In Journal of Cognitive Neuroscience, 29:10 we collect contributed articles that seek to move the conversation beyond specific functions of subregions of pFC, focusing instead on general roles that support pFC involvement in a wide variety of behaviors and across a variety of experimental paradigms.