A neuro-computational model of economic decisions

The influence of reward and loss outcomes after free- and forced-tasks on voluntary task choice

In four experiments, we investigated the impact of outcomes and processing mode (free versus forced) on subsequent voluntary task-switching behavior. Participants freely chose between two tasks or were forced to perform one, and the feedback they received randomly varied after correct performance (reward or no-reward; loss or no-loss). In general, we reasoned that the most recently applied task goal is usually the most valued one, leading people to prefer task repetitions over switches. However, the task values might be additionally biased by previous outcomes and the previous processing mode. Indeed, negatively reinforcing tasks with no-reward or losses generally resulted in more subsequent switches. Additionally, participants demonstrated a stronger attachment to free- compared to forced-tasks, as indicated by more switches when the previous task was forced, suggesting that people generally value free over forced-choice task goals. Moreover, the reward manipulation had a greater influence on switching behavior following free- compared to forced-tasks in Exp. 1 and Exp. 3, suggesting a stronger emphasis on evaluating rewarding outcomes associated with free-task choices. However, this inflationary effect on task choice seemed to be limited to reward and situations where task choice and performance more strongly overlap. Specifically, there was no evidence that switching behavior was differentially influenced after free-and forced-task as a function of losses (Exp. 2) or reward when task choice and task performance were separated (Exp. 4). Overall, the results provide new insights into how the valuation of task goals based on choice freedom and outcome feedback can influence voluntary task choices.

Orbitofrontal high-gamma reflects spike-dissociable value and decision mechanisms

The orbitofrontal cortex (OFC) plays a crucial role in value-based decision-making. While previous research has focused on spiking activity in OFC neurons, the role of OFC local field potentials (LFPs) in decision-making remains unclear. LFPs are important because they can reflect synaptic and subthreshold activity not directly coupled to spiking, and because they are potential targets for less invasive forms of brain-machine interface (BMI). We recorded LFPs and spiking activity using multi-channel vertical probes while monkeys performed a two-option value-based decision-making task. We compared the value- and decision-coding properties of high-gamma range LFPs (HG, 50-150 Hz) to the coding properties of spiking multi-unit activity (MUA) recorded concurrently on the same electrodes. Results show that HG and MUA both represent the values of decision targets, and that their representations have similar temporal profiles in a trial. However, we also identified value-coding properties of HG that were dissociable from the concurrently-measured MUA. On average across channels, HG amplitude increased monotonically with value, whereas the average value encoding in MUA was net neutral. HG also encoded a signal consistent with a comparison between the values of the two targets, a signal which was much weaker in MUA. In individual channels, HG was better able to predict choice outcomes than MUA; however, when simultaneously recorded channels were combined in population-based decoder, MUA provided more accurate predictions than HG. Interestingly, HG value representations were accentuated in channels in or near shallow cortical layers, suggesting a dissociation between neuronal sources of HG and MUA. In summary, we find that HG signals are dissociable from MUA with respect to cognitive variables encoded in prefrontal cortex, evident in the monotonic encoding of value, stronger encoding of value comparisons, and more accurate predictions about behavior. High-frequency LFPs may therefore be a viable - or even preferable - target for BMIs to assist cognitive function, opening the possibility for less invasive access to mental contents that would otherwise be observable only with spike-based measures.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Risky decisions are influenced by individual attributes as a function of risk preference

It has long been assumed in economic theory that multi-attribute decisions involving several attributes or dimensions - such as probabilities and amounts of money to be earned during risky choices - are resolved by first combining the attributes of each option to form an overall expected value and then comparing the expected values of the alternative options, using a unique evidence accumulation process. A plausible alternative would be performing independent comparisons between the individual attributes and then integrating the results of the comparisons afterwards. Here, we devise a novel method to disambiguate between these types of models, by orthogonally manipulating the expected value of choice options and the relative salience of their attributes. Our results, based on behavioral measures and drift-diffusion models, provide evidence in favor of the framework where information about individual attributes independently impacts deliberation. This suggests that risky decisions are resolved by running in parallel multiple comparisons between the separate attributes - possibly alongside an additional comparison of expected value. This result stands in contrast with the assumption of standard economic theory that choices require a unique comparison of expected values and suggests that at the cognitive level, decision processes might be more distributed than commonly assumed. Beyond our planned analyses, we also discovered that attribute salience affects people of different risk preference type in different ways: risk-averse participants seem to focus more on probability, except when monetary amount is particularly high; risk-neutral/seeking participants, in contrast, seem to focus more on monetary amount, except when probability is particularly low.

Multi-attribute decision-making in macaques relies on direct attribute comparisons

In value-based decisions, there are frequently multiple attributes, such as cost, quality, or quantity, that contribute to the overall goodness of an option. Since one option may not be better in all attributes at once, the decision process should include a means of weighing relevant attributes. Most decision-making models solve this problem by computing an integrated value, or utility, for each option from a weighted combination of attributes. However, behavioral anomalies in decision-making, such as context effects, indicate that other attribute-specific computations might be taking place. Here, we tested whether rhesus macaques show evidence of attribute-specific processing in a value-based decision-making task. Monkeys made a series of decisions involving choice options comprising a sweetness and probability attribute. Each attribute was represented by a separate bar with one of two mappings between bar size and the magnitude of the attribute (i.e., bigger=better or bigger=worse). We found that translating across different mappings produced selective impairments in decision-making. When like attributes differed, monkeys were prevented from easily making direct attribute comparisons, and choices were less accurate and preferences were more variable. This was not the case when mappings of unalike attributes within the same option were different. Likewise, gaze patterns favored transitions between like attributes over transitions between unalike attributes of the same option, so that like attributes were sampled sequentially to support within-attribute comparisons. Together, these data demonstrate that value-based decisions rely, at least in part, on directly comparing like attributes of multi-attribute options.

Significance Statement

Value-based decision-making is a cognitive function impacted by a number of clinical conditions, including substance use disorder and mood disorders. Understanding the neural mechanisms, including online processing steps involved in decision formation, will provide critical insights into decision-making deficits characteristic of human psychiatric disorders. Using rhesus monkeys as a model species capable of complex decision-making, this study shows that decisions involve a process of comparing like features, or attributes, of multi-attribute options. This is contrary to popular models of decision-making in which attributes are first combined into an overall value, or utility, to make a choice. Therefore, these results serve as an important foundation for establishing a more complete understanding of the neural mechanisms involved in forming complex decisions.

Action-rule-based cognitive control enables efficient execution of stimulus-response conflict tasks: a model validation of Simon task performance

Introduction

The human brain can flexibly modify behavioral rules to optimize task performance (speed and accuracy) by minimizing cognitive load. To show this flexibility, we propose an action-rule-based cognitive control (ARC) model. The ARC model was based on a stochastic framework consistent with an active inference of the free energy principle, combined with schematic brain network systems regulated by the dorsal anterior cingulate cortex (dACC), to develop several hypotheses for demonstrating the validity of the ARC model.

Methods

A step-motion Simon task was developed involving congruence or incongruence between important symbolic information (illustration of a foot labeled "L" or "R," where "L" requests left and "R" requests right foot movement) and irrelevant spatial information (whether the illustration is actually of a left or right foot). We made predictions for behavioral and brain responses to testify to the theoretical predictions.

Results

Task responses combined with event-related deep-brain activity (ER-DBA) measures demonstrated a key contribution of the dACC in this process and provided evidence for the main prediction that the dACC could reduce the Shannon surprise term in the free energy formula by internally reversing the irrelevant rapid anticipatory postural adaptation. We also found sequential effects with modulated dip depths of ER-DBA waveforms that support the prediction that repeated stimuli with the same congruency can promote remodeling of the internal model through the information gain term while counterbalancing the surprise term.

Discussion

Overall, our results were consistent with experimental predictions, which may support the validity of the ARC model. The sequential effect accompanied by dip modulation of ER-DBA waveforms suggests that cognitive cost is saved while maintaining cognitive performance in accordance with the framework of the ARC based on 1-bit congruency-dependent selective control.

Working memory and attention in choice

We study the role of attention and working memory in choices where options are presented sequentially rather than simultaneously. We build a model where a costly attention effort is chosen, which can vary over time. Evidence is accumulated proportionally to this effort and the utility of the reward. Crucially, the evidence accumulated decays over time. Optimal attention allocation maximizes expected utility from final choice; the optimal solution takes the decay into account, so attention is preferentially devoted to later times; but convexity of the flow attention cost prevents it from being concentrated near the end. We test this model with a choice experiment where participants observe sequentially two options. In our data the option presented first is, everything else being equal, significantly less likely to be chosen. This recency effect has a natural explanation with appropriate parameter values in our model of leaky evidence accumulation, where the decline is stronger for the option observed first. Analysis of choice, response time and brain imaging data provide support for the model. Working memory plays an essential role. The recency bias is stronger for participants with weaker performance in working memory tasks. Also activity in parietal areas, coding the stored value in working, declines over time as predicted.

Value dynamics affect choice preparation during decision-making

During decision-making, neurons in the orbitofrontal cortex (OFC) sequentially represent the value of each option in turn, but it is unclear how these dynamics are translated into a choice response. One brain region that may be implicated in this process is the anterior cingulate cortex (ACC), which strongly connects with OFC and contains many neurons that encode the choice response. We investigated how OFC value signals interacted with ACC neurons encoding the choice response by performing simultaneous high-channel count recordings from the two areas in nonhuman primates. ACC neurons encoding the choice response steadily increased their firing rate throughout the decision-making process, peaking shortly before the time of the choice response. Furthermore, the value dynamics in OFC affected ACC ramping-when OFC represented the more valuable option, ACC ramping accelerated. Because OFC tended to represent the more valuable option more frequently and for a longer duration, this interaction could explain how ACC selects the more valuable response.

Experiential values are underweighted in decisions involving symbolic options

Standard models of decision-making assume each option is associated with subjective value, regardless of whether this value is inferred from experience (experiential) or explicitly instructed probabilistic outcomes (symbolic). In this study, we present results that challenge the assumption of unified representation of experiential and symbolic value. Across nine experiments, we presented participants with hybrid decisions between experiential and symbolic options. Participants' choices exhibited a pattern consistent with a systematic neglect of the experiential values. This normatively irrational decision strategy held after accounting for alternative explanations, and persisted even when it bore an economic cost. Overall, our results demonstrate that experiential and symbolic values are not symmetrically considered in hybrid decisions, suggesting they recruit different representational systems that may be assigned different priority levels in the decision process. These findings challenge the dominant models commonly used in value-based decision-making research.

The role of the lateral orbitofrontal cortex in creating cognitive maps

We use mental models of the world-cognitive maps-to guide behavior. The lateral orbitofrontal cortex (lOFC) is typically thought to support behavior by deploying these maps to simulate outcomes, but recent evidence suggests that it may instead support behavior by underlying map creation. We tested between these two alternatives using outcome-specific devaluation and a high-potency chemogenetic approach. Selectively inactivating lOFC principal neurons when male rats learned distinct cue-outcome associations, but before outcome devaluation, disrupted subsequent inference, confirming a role for the lOFC in creating new maps. However, lOFC inactivation surprisingly led to generalized devaluation, a result that is inconsistent with a complete mapping failure. Using a reinforcement learning framework, we show that this effect is best explained by a circumscribed deficit in credit assignment precision during map construction, suggesting that the lOFC has a selective role in defining the specificity of associations that comprise cognitive maps.

Value representations in the rodent orbitofrontal cortex drive learning, not choice

Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice: the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning: expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here, we employ a recently developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.

Orbitofrontal cortex contributes to the comparison of values underlying economic choices

Economic choices between goods entail the computation and comparison of subjective values. Previous studies examined neuronal activity in the orbitofrontal cortex (OFC) of monkeys choosing between different types of juices. Three groups of neurons were identified: offer value cells encoding the value of individual offers, chosen juice cells encoding the identity of the chosen juice, and chosen value cells encoding the value of the chosen offer. The encoded variables capture both the input (offer value) and the output (chosen juice, chosen value) of the decision process, suggesting that values are compared within OFC. Recent work demonstrates that choices are causally linked to the activity of offer value cells. Conversely, the hypothesis that OFC contributes to value comparison has not been confirmed. Here we show that weak electrical stimulation of OFC specifically disrupts value comparison without altering offer values. This result implies that neuronal populations in OFC participate in value comparison.

Taking stock of value in the orbitofrontal cortex

People with damage to the orbitofrontal cortex (OFC) have specific problems making decisions, whereas their other cognitive functions are spared. Neurophysiological studies have shown that OFC neurons fire in proportion to the value of anticipated outcomes. Thus, a central role of the OFC is to guide optimal decision-making by signalling values associated with different choices. Until recently, this view of OFC function dominated the field. New data, however, suggest that the OFC may have a much broader role in cognition by representing cognitive maps that can be used to guide behaviour and that value is just one of many variables that are important for behavioural control. In this Review, we critically evaluate these two alternative accounts of OFC function and examine how they might be reconciled.

Logistic analysis of choice data: A primer

Logistic regressions were developed in economics to model individual choice behavior. In recent years, they have become an important tool in decision neuroscience. Here, I describe and discuss different logistic models, emphasizing the underlying assumptions and possible interpretations. Logistic models may be used to quantify a variety of behavioral traits, including the relative subjective value of different goods, the choice accuracy, risk attitudes, and choice biases. More complex logistic models can be used for choices between good bundles, in cases of nonlinear value functions, and for choices between multiple options. Finally, logistic models can quantify the explanatory power of neuronal activity on choices, thus providing a valid alternative to receiver operating characteristic (ROC) analyses.

Economic Choices under Simultaneous or Sequential Offers Rely on the Same Neural Circuit

A series of studies in which monkeys chose between two juices offered in variable amounts identified in the orbitofrontal cortex (OFC) different groups of neurons encoding the value of individual options (offer value), the binary choice outcome (chosen juice), and the chosen value. These variables capture both the input and the output of the choice process, suggesting that the cell groups identified in OFC constitute the building blocks of a decision circuit. Several lines of evidence support this hypothesis. However, in previous experiments offers were presented simultaneously, raising the question of whether current notions generalize to when goods are presented or are examined in sequence. Recently, Ballesta and Padoa-Schioppa (2019) examined OFC activity under sequential offers. An analysis of neuronal responses across time windows revealed that a small number of cell groups encoded specific sequences of variables. These sequences appeared analogous to the variables identified under simultaneous offers, but the correspondence remained tentative. Thus, in the present study, we examined the relation between cell groups found under sequential versus simultaneous offers. We recorded from the OFC while monkeys chose between different juices. Trials with simultaneous and sequential offers were randomly interleaved in each session. We classified cells in each choice modality, and we examined the relation between the two classifications. We found a strong correspondence; in other words, the cell groups measured under simultaneous offers and under sequential offers were one and the same. This result indicates that economic choices under simultaneous or sequential offers rely on the same neural circuit.SIGNIFICANCE STATEMENT Research in the past 20 years has shed light on the neuronal underpinnings of economic choices. A large number of results indicates that decisions between goods are formed in a neural circuit within the orbitofrontal cortex. In most previous studies, subjects chose between two goods offered simultaneously. Yet, in daily situations, goods available for choice are often presented or examined in sequence. Here we recorded neuronal activity in the primate orbitofrontal cortex alternating trials under simultaneous and under sequential offers. Our analyses demonstrate that the same neural circuit supports choices in the two modalities. Hence, current notions on the neuronal mechanisms underlying economic decisions generalize to choices under sequential offers.

Neuronal origins of reduced accuracy and biases in economic choices under sequential offers

Economic choices are characterized by a variety of biases. Understanding their origins is a long-term goal for neuroeconomics, but progress on this front has been limited. Here, we examined choice biases observed when two goods are offered sequentially. In the experiments, rhesus monkeys chose between different juices offered simultaneously or in sequence. Choices under sequential offers were less accurate (higher variability). They were also biased in favor of the second offer (order bias) and in favor of the preferred juice (preference bias). Analysis of neuronal activity recorded in the orbitofrontal cortex revealed that these phenomena emerged at different computational stages. Lower choice accuracy reflected weaker offer value signals (valuation stage), the order bias emerged during value comparison (decision stage), and the preference bias emerged late in the trial (post-comparison). By neuronal measures, each phenomenon reduced the value obtained on average in each trial and was thus costly to the monkey.

The Neural Instantiation of an Abstract Cognitive Map for Economic Choice

Since the discovery of cognitive maps in rodent hippocampus (HC), the cognitive map has evolved from originally referring to spatial representations encoding locations and objects in Euclidean spaces to a general low-dimensional organization of information along selected feature dimensions. A cognitive map includes hypothetical constructs that bridge between environmental stimuli and the final overt behavior. To neuroeconomists, utility and utility functions are such constructs with neurobiological basis that drive choice behavior. Emergence of distinct functional neuron groups in the primate orbitofrontal cortex (OFC) during simple economic choice indicates the formation of an abstract cognitive map for organizing information of goods for value computation. Experimental evidence suggests that organization of neuronal activity in such cognitive map reflects the abstraction of core task features. Thus, such map can be adapted to accommodate economic choices under various task contexts.

The hierarchical construction of value

Here we argue that the assignment of subjective value to potential outcomes at the time of decision-making is an active process, in which individual features of a potential outcome of varying degrees of abstraction are represented hierarchically and integrated in a weighted fashion to produce an overall value judgment. We implicate the lateral orbital and medial prefrontal cortex in this function, situating these areas more broadly within a hierarchical integration process that takes place throughout the cortex for the ultimate purpose of valuing options to guide decisions.

Frontal circuit specialisations for decision making

There is widespread consensus that distributed circuits across prefrontal and anterior cingulate cortex (PFC/ACC) are critical for reward-based decision making. The circuit specialisations of these areas in primates were likely shaped by their foraging niche, in which decision making is typically sequential, attention-guided and temporally extended. Here, I argue that in humans and other primates, PFC/ACC circuits are functionally specialised in two ways. First, microcircuits found across PFC/ACC are highly recurrent in nature and have synaptic properties that support persistent activity across temporally extended cognitive tasks. These properties provide the basis of a computational account of time-varying neural activity within PFC/ACC as a decision is being made. Second, the macrocircuit connections (to other brain areas) differ between distinct PFC/ACC cytoarchitectonic subregions. This variation in macrocircuit connections explains why PFC/ACC subregions make unique contributions to reward-based decision tasks and how these contributions are shaped by attention. They predict dissociable neural representations to emerge in orbitofrontal, anterior cingulate and dorsolateral prefrontal cortex during sequential attention-guided choice, as recently confirmed in neurophysiological recordings.

Partial integration of the components of value in anterior cingulate cortex

Evaluation often involves integrating multiple determinants of value, such as the different possible outcomes in risky choice. A brain region can be placed either before or after a presumed evaluation stage by measuring how responses of its neurons depend on multiple determinants of value. A brain region could also, in principle, show partial integration, which would indicate that it occupies a middle position between (preevaluative) nonintegration and (postevaluative) full integration. Existing mathematical techniques cannot distinguish full from partial integration and therefore risk misidentifying regional function. Here we use a new Bayesian regression-based approach to analyze responses of neurons in dorsal anterior cingulate cortex (dACC) to risky offers. We find that dACC neurons only partially integrate across outcome dimensions, indicating that dACC cannot be assigned to either a pre- or postevaluative position. Neurons in dACC also show putative signatures of value comparison, thereby demonstrating that comparison does not require complete evaluation before proceeding. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Is value-based choice repetition susceptible to medial frontal transcranial direct current stimulation (tDCS)? A preregistered study

In value-based decision making, people have to weigh different options based on their subjective value. This process, however, also is influenced by choice biases, such as choice repetition: in a series of choices, people are more likely to repeat their decision than to switch to a different choice. Previously, it was shown that transcranial direct current stimulation (tDCS) can affect such choice biases. We applied tDCS over the medial prefrontal cortex to investigate whether tDCS can alter choice repetition in value-based decision making. In a preregistered study, we applied anodal, cathodal, and sham tDCS stimulation to 52 participants. While we found robust choice repetition effects, we did not find support for an effect of tDCS stimulation. We discuss these findings within the larger scope of the tDCS literature and highlight the potential roles of interindividual variability and current density strength.

Excitatory-inhibitory tone shapes decision strategies in a hierarchical neural network model of multi-attribute choice

We are constantly faced with decisions between alternatives defined by multiple attributes, necessitating an evaluation and integration of different information sources. Time-varying signals in multiple brain areas are implicated in decision-making; but we lack a rigorous biophysical description of how basic circuit properties, such as excitatory-inhibitory (E/I) tone and cascading nonlinearities, shape attribute processing and choice behavior. Furthermore, how such properties govern choice performance under varying levels of environmental uncertainty is unknown. We investigated two-attribute, two-alternative decision-making in a dynamical, cascading nonlinear neural network with three layers: an input layer encoding choice alternative attribute values; an intermediate layer of modules processing separate attributes; and a final layer producing the decision. Depending on intermediate layer E/I tone, the network displays distinct regimes characterized by linear (I), convex (II) or concave (III) choice indifference curves. In regimes I and II, each option's attribute information is additively integrated. In regime III, time-varying nonlinear operations amplify the separation between offer distributions by selectively attending to the attribute with the larger differences in input values. At low environmental uncertainty, a linear combination most consistently selects higher valued alternatives. However, at high environmental uncertainty, regime III is more likely than a linear operation to select alternatives with higher value. Furthermore, there are conditions where readout from the intermediate layer could be experimentally indistinguishable from the final layer. Finally, these principles are used to examine multi-attribute decisions in systems with reduced inhibitory tone, leading to predictions of different choice patterns and overall performance between those with restrictions on inhibitory tone and neurotypicals.

Prefrontal cortex represents heuristics that shape choice bias and its integration into future behavior

Goal-directed behavior requires integrating sensory information with prior knowledge about the environment. Behavioral biases that arise from these priors could increase positive outcomes when the priors match the true structure of the environment, but mismatches also happen frequently and could cause unfavorable outcomes. Biases that reduce gains and fail to vanish with training indicate fundamental suboptimalities arising from ingrained heuristics of the brain. Here, we report systematic, gain-reducing choice biases in highly trained monkeys performing a motion direction discrimination task where only the current stimulus is behaviorally relevant. The monkey's bias fluctuated at two distinct time scales: slow, spanning tens to hundreds of trials, and fast, arising from choices and outcomes of the most recent trials. Our findings enabled single trial prediction of biases, which influenced the choice especially on trials with weak stimuli. The pre-stimulus activity of neuronal ensembles in the monkey prearcuate gyrus represented these biases as an offset along the decision axis in the state space. This offset persisted throughout the stimulus viewing period, when sensory information was integrated, leading to a biased choice. The pre-stimulus representation of history-dependent bias was functionally indistinguishable from the neural representation of upcoming choice before stimulus onset, validating our model of single-trial biases and suggesting that pre-stimulus representation of choice could be fully defined by biases inferred from behavioral history. Our results indicate that the prearcuate gyrus reflects intrinsic heuristics that compute bias signals, as well as the mechanisms that integrate them into the oculomotor decision-making process.

Values encoded in orbitofrontal cortex are causally related to economic choices

In the eighteenth century, Daniel Bernoulli, Adam Smith and Jeremy Bentham proposed that economic choices rely on the computation and comparison of subjective values1. This hypothesis continues to inform modern economic theory2 and research in behavioural economics3, but behavioural measures are ultimately not sufficient to verify the proposal4. Consistent with the hypothesis, when agents make choices, neurons in the orbitofrontal cortex (OFC) encode the subjective value of offered and chosen goods5. Value-encoding cells integrate multiple dimensions6-9, variability in the activity of each cell group correlates with variability in choices10,11 and the population dynamics suggests the formation of a decision12. However, it is unclear whether these neural processes are causally related to choices. More generally, the evidence linking economic choices to value signals in the brain13-15 remains correlational16. Here we show that neuronal activity in the OFC is causal to economic choices. We conducted two experiments using electrical stimulation in rhesus monkeys (Macaca mulatta). Low-current stimulation increased the subjective value of individual offers and thus predictably biased choices. Conversely, high-current stimulation disrupted both the computation and the comparison of subjective values, and thus increased choice variability. These results demonstrate a causal chain linking subjective values encoded in OFC to valuation and choice.

Neural mechanisms of economic choices in mice

Economic choices entail computing and comparing subjective values. Evidence from primates indicates that this behavior relies on the orbitofrontal cortex. Conversely, previous work in rodents provided conflicting results. Here we present a mouse model of economic choice behavior, and we show that the lateral orbital (LO) area is intimately related to the decision process. In the experiments, mice chose between different juices offered in variable amounts. Choice patterns closely resembled those measured in primates. Optogenetic inactivation of LO dramatically disrupted choices by inducing erratic changes of relative value and by increasing choice variability. Neuronal recordings revealed that different groups of cells encoded the values of individual options, the binary choice outcome and the chosen value. These groups match those previously identified in primates, except that the neuronal representation in mice is spatial (in monkeys it is good-based). Our results lay the foundations for a circuit-level analysis of economic decisions.

The Transition from Evaluation to Selection Involves Neural Subspace Reorganization in Core Reward Regions

Economic choice proceeds from evaluation, in which we contemplate options, to selection, in which we weigh options and choose one. These stages must be differentiated so that decision makers do not proceed to selection before evaluation is complete. We examined responses of neurons in two core reward regions, orbitofrontal (OFC) and ventromedial prefrontal cortex (vmPFC), during two-option choice with asynchronous offer presentation. Our data suggest that neurons selective during the first (presumed evaluation) and second (presumed comparison and selection) offer epochs come from a single pool. Stage transition is accompanied by a shift toward orthogonality in the low-dimensional population response manifold. Nonetheless, the relative position of each option in driving responses in the population subspace is preserved. The orthogonalization we observe supports the hypothesis that the transition from evaluation to selection leads to reorganization of response subspace and suggests a mechanism by which value-related signals are prevented from prematurely driving choice.

Excitatory and Inhibitory Subnetworks Are Equally Selective during Decision-Making and Emerge Simultaneously during Learning

Inhibitory neurons, which play a critical role in decision-making models, are often simplified as a single pool of non-selective neurons lacking connection specificity. This assumption is supported by observations in the primary visual cortex: inhibitory neurons are broadly tuned in vivo and show non-specific connectivity in slice. The selectivity of excitatory and inhibitory neurons within decision circuits and, hence, the validity of decision-making models are unknown. We simultaneously measured excitatory and inhibitory neurons in the posterior parietal cortex of mice judging multisensory stimuli. Surprisingly, excitatory and inhibitory neurons were equally selective for the animal's choice, both at the single-cell and population level. Further, both cell types exhibited similar changes in selectivity and temporal dynamics during learning, paralleling behavioral improvements. These observations, combined with modeling, argue against circuit architectures assuming non-selective inhibitory neurons. Instead, they argue for selective subnetworks of inhibitory and excitatory neurons that are shaped by experience to support expert decision-making.

Economic Decisions through Circuit Inhibition

Economic choices between goods are thought to rely on the orbitofrontal cortex (OFC), but the decision mechanisms remain poorly understood. To shed light on this fundamental issue, we recorded from the OFC of monkeys choosing between two juices offered sequentially. An analysis of firing rates across time windows revealed the presence of different groups of neurons similar to those previously identified under simultaneous offers. This observation suggested that economic decisions in the two modalities are formed in the same neural circuit. We then examined several hypotheses on the decision mechanisms. OFC neurons encoded good identities and values in a juice-based representation (labeled lines). Contrary to previous assessments, our data argued against the idea that decisions rely on mutual inhibition at the level of offer values. In fact, we showed that previous arguments for mutual inhibition were confounded by differences in value ranges. Instead, decisions seemed to involve mechanisms of circuit inhibition, whereby each offer value indirectly inhibited neurons encoding the opposite choice outcome. Our results reconcile a variety of previous findings and provide a general account for the neuronal underpinnings of economic choices.

Categorical encoding of decision variables in orbitofrontal cortex

A fundamental and recurrent question in systems neuroscience is that of assessing what variables are encoded by a given population of neurons. Such assessments are often challenging because neurons in one brain area may encode multiple variables, and because neuronal representations might be categorical or non-categorical. These issues are particularly pertinent to the representation of decision variables in the orbitofrontal cortex (OFC)-an area implicated in economic choices. Here we present a new algorithm to assess whether a neuronal representation is categorical or non-categorical, and to identify the encoded variables if the representation is indeed categorical. The algorithm is based on two clustering procedures, one variable-independent and the other variable-based. The two partitions are then compared through adjusted mutual information. The present algorithm overcomes limitations of previous approaches and is widely applicable. We tested the algorithm on synthetic data and then used it to examine neuronal data recorded in the primate OFC during economic decisions. Confirming previous assessments, we found the neuronal representation in OFC to be categorical in nature. We also found that neurons in this area encode the value of individual offers, the binary choice outcome and the chosen value. In other words, during economic choice, neurons in the primate OFC encode decision variables in a categorical way.

Perceptual Decision-Making: A Field in the Midst of a Transformation

Major changes are underway in the field of perceptual decision-making. Single-neuron studies have given way to population recordings with identified cell types, traditional analyses have been extended to accommodate these large and diverse collections of neurons, and novel methods of neural disruption have provided insights about causal circuits. Further, the field has expanded to include multiple new species: rodents and invertebrates, for example, have been instrumental in demonstrating the importance of internal state on neural responses. Finally, a renewed interest in ethological stimuli prompted development of new behaviors, frequently analyzed by new, automated movement tracking methods. Taken together, these advances constitute a seismic shift in both our approach and understanding of how incoming sensory signals are used to guide decisions.

Economic Choice as an Untangling of Options into Actions

We propose that economic choice can be understood as a gradual transformation from a domain of options to one of the actions. We draw an analogy with the idea of untangling information in the form vision system and propose that form vision and economic choice may be two aspects of a larger process that sculpts actions based on sensory inputs. From this viewpoint, choice results from the accumulated effect of repetitions of simple computations. These may consist primarily of relative valuations (evaluations relative to the value of rejection, perhaps in a manner akin to divisive normalization) applied to individual offers. With regard to economic choice, cortical brain regions differ primarily in their position and in what information they prioritize, and do not-with a few exceptions-have categorically distinct roles. Each region's specific contribution is determined largely by its inputs; thus, understanding connectivity is crucial for understanding choice. This view suggests that there is no single site of choice, that there is no meaningful distinction between pre- and post-decisionality, and that there is no explicit representation of value in the brain.

Robust mixture modeling reveals category-free selectivity in reward region neuronal ensembles

Classification of neurons into clusters based on their response properties is an important tool for gaining insight into neural computations. However, it remains unclear to what extent neurons fall naturally into discrete functional categories. We developed a Bayesian method that models the tuning properties of neural populations as a mixture of multiple types of task-relevant response patterns. We applied this method to data from several cortical and striatal regions in economic choice tasks. In all cases, neurons fell into only two clusters: one multiple-selectivity cluster containing all cells driven by task variables of interest and another of no selectivity for those variables. The single cluster of task-sensitive cells argues against robust categorical tuning in these areas. The no-selectivity cluster was unanticipated and raises important questions about what distinguishes these neurons and what role they play. Moreover, the ability to formally identify these nonselective cells allows for more accurate measurement of ensemble effects by excluding or appropriately down-weighting them in analysis. Our findings provide a valuable tool for analysis of neural data, challenge simple categorization schemes previously proposed for these regions, and place useful constraints on neurocomputational models of economic choice and control. NEW & NOTEWORTHY We present a Bayesian method for formally detecting whether a population of neurons can be naturally classified into clusters based on their response tuning properties. We then examine several data sets of reward system neurons for variables and find in all cases that neurons can be classified into only two categories: a functional class and a non-task-driven class. These results provide important constraints for neural models of the reward system.

Correlates of economic decisions in the dorsal and subgenual anterior cingulate cortices

The anterior cingulate cortex can be divided into distinct ventral (subgenual, sgACC) and dorsal (dACC), portions. The role of dACC in value-based decision-making is hotly debated, while the role of sgACC is poorly understood. We recorded neuronal activity in both regions in rhesus macaques performing a token-gambling task. We find that both encode many of the same variables; including integrated offered values of gambles, primary as well as secondary reward outcomes, number of current tokens and anticipated rewards. Both regions exhibit memory traces for offer values and putative value comparison signals. Both regions use a consistent scheme to encode the value of the attended option. This result suggests that neurones do not appear to be specialized for specific offers (that is, neurones use an attentional as opposed to labelled line coding scheme). We also observed some differences between the two regions: (i) coding strengths in dACC were consistently greater than those in sgACC, (ii) neurones in sgACC responded especially to losses and in anticipation of primary rewards, while those in dACC showed more balanced responding and (iii) responses to the first offer were slightly faster in sgACC. These results indicate that sgACC and dACC have some functional overlap in economic choice, and are consistent with the idea, inspired by neuroanatomy, which sgACC may serve as input to dACC.

Using a Simple Neural Network to Delineate Some Principles of Distributed Economic Choice

The brain uses a mixture of distributed and modular organization to perform computations and generate appropriate actions. While the principles under which the brain might perform computations using modular systems have been more amenable to modeling, the principles by which the brain might make choices using distributed principles have not been explored. Our goal in this perspective is to delineate some of those distributed principles using a neural network method and use its results as a lens through which to reconsider some previously published neurophysiological data. To allow for direct comparison with our own data, we trained the neural network to perform binary risky choices. We find that value correlates are ubiquitous and are always accompanied by non-value information, including spatial information (i.e., no pure value signals). Evaluation, comparison, and selection were not distinct processes; indeed, value signals even in the earliest stages contributed directly, albeit weakly, to action selection. There was no place, other than at the level of action selection, at which dimensions were fully integrated. No units were specialized for specific offers; rather, all units encoded the values of both offers in an anti-correlated format, thus contributing to comparison. Individual network layers corresponded to stages in a continuous rotation from input to output space rather than to functionally distinct modules. While our network is likely to not be a direct reflection of brain processes, we propose that these principles should serve as hypotheses to be tested and evaluated for future studies.

A neural network model for the orbitofrontal cortex and task space acquisition during reinforcement learning

Reinforcement learning has been widely used in explaining animal behavior. In reinforcement learning, the agent learns the value of the states in the task, collectively constituting the task state space, and uses the knowledge to choose actions and acquire desired outcomes. It has been proposed that the orbitofrontal cortex (OFC) encodes the task state space during reinforcement learning. However, it is not well understood how the OFC acquires and stores task state information. Here, we propose a neural network model based on reservoir computing. Reservoir networks exhibit heterogeneous and dynamic activity patterns that are suitable to encode task states. The information can be extracted by a linear readout trained with reinforcement learning. We demonstrate how the network acquires and stores task structures. The network exhibits reinforcement learning behavior and its aspects resemble experimental findings of the OFC. Our study provides a theoretical explanation of how the OFC may contribute to reinforcement learning and a new approach to understanding the neural mechanism underlying reinforcement learning.

A neuronal theory of sequential economic choice

Results of recent studies point towards a new framework for the neural bases of economic choice. The principles of this framework include the idea that evaluation is limited to a single option within the focus of attention and that we accept or reject that option relative to the entire set of alternatives. Rejection leads attention to a new option, although it can later switch back to a previously rejected one. The option to which a neuron's firing rate refers is determined dynamically by attention and not stably by labelled lines. Value is always computed relative to the value of rejection. Comparison results not from explicit competition between discrete populations of neurons, but indirectly, as in a horse race, from the fact that the first option whose value crosses a threshold is selected. Consequently, comparison can occur within a single pool of neurons rather than by competition between two or more neuronal populations. The computations that constitute comparison thus occur at multiple levels, including premotor levels, simultaneously (i.e. the brain uses a distributed consensus), and not in discrete stages. This framework suggests a solution to a set of otherwise unresolved neuronal binding problems that result from the need to link options to values, comparisons to actions, and choices to outcomes.

Correlates of decisional dynamics in the dorsal anterior cingulate cortex

We hypothesized that during binary economic choice, decision makers use the first option they attend as a default to which they compare the second. To test this idea, we recorded activity of neurons in the dorsal anterior cingulate cortex (dACC) of macaques choosing between gambles presented asynchronously. We find that ensemble encoding of the value of the first offer includes both choice-dependent and choice-independent aspects, as if reflecting a partial decision. That is, its responses are neither entirely pre- nor post-decisional. In contrast, coding of the value of the second offer is entirely decision dependent (i.e., post-decisional). This result holds even when offer-value encodings are compared within the same time period. Additionally, we see no evidence for 2 pools of neurons linked to the 2 offers; instead, all comparison appears to occur within a single functionally homogenous pool of task-selective neurons. These observations suggest that economic choices reflect a context-dependent evaluation of attended options. Moreover, they raise the possibility that value representations reflect, to some extent, a tentative commitment to a choice.

Orbitofrontal Cortex: A Neural Circuit for Economic Decisions

Economic choice behavior entails the computation and comparison of subjective values. A central contribution of neuroeconomics has been to show that subjective values are represented explicitly at the neuronal level. With this result at hand, the field has increasingly focused on the difficult question of where in the brain and how exactly subjective values are compared to make a decision. Here, we review a broad range of experimental and theoretical results suggesting that good-based decisions are generated in a neural circuit within the orbitofrontal cortex (OFC). The main lines of evidence supporting this proposal include the fact that goal-directed behavior is specifically disrupted by OFC lesions, the fact that different groups of neurons in this area encode the input and the output of the decision process, the fact that activity fluctuations in each of these cell groups correlate with choice variability, and the fact that these groups of neurons are computationally sufficient to generate decisions. Results from other brain regions are consistent with the idea that good-based decisions take place in OFC and indicate that value signals inform a variety of mental functions. We also contrast the present proposal with other leading models for the neural mechanisms of economic decisions. Finally, we indicate open questions and suggest possible directions for future research.

Optimal coding and neuronal adaptation in economic decisions

During economic decisions, offer value cells in orbitofrontal cortex (OFC) encode the values of offered goods. Furthermore, their tuning functions adapt to the range of values available in any given context. A fundamental and open question is whether range adaptation is behaviorally advantageous. Here we present a theory of optimal coding for economic decisions. We propose that the representation of offer values is optimal if it ensures maximal expected payoff. In this framework, we examine offer value cells in non-human primates. We show that their responses are quasi-linear even when optimal tuning functions are highly non-linear. Most importantly, we demonstrate that for linear tuning functions range adaptation maximizes the expected payoff. Thus value coding in OFC is functionally rigid (linear tuning) but parametrically plastic (range adaptation with optimal gain). Importantly, the benefit of range adaptation outweighs the cost of functional rigidity. While generally suboptimal, linear tuning may facilitate transitive choices.

The Attentional Drift Diffusion Model of Simple Perceptual Decision-Making

Perceptual decisions requiring the comparison of spatially distributed stimuli that are fixated sequentially might be influenced by fluctuations in visual attention. We used two psychophysical tasks with human subjects to investigate the extent to which visual attention influences simple perceptual choices, and to test the extent to which the attentional Drift Diffusion Model (aDDM) provides a good computational description of how attention affects the underlying decision processes. We find evidence for sizable attentional choice biases and that the aDDM provides a reasonable quantitative description of the relationship between fluctuations in visual attention, choices and reaction times. We also find that exogenous manipulations of attention induce choice biases consistent with the predictions of the model.

A distributed, hierarchical and recurrent framework for reward-based choice

Many accounts of reward-based choice argue for distinct component processes that are serial and functionally localized. In this Opinion article, we argue for an alternative viewpoint, in which choices emerge from repeated computations that are distributed across many brain regions. We emphasize how several features of neuroanatomy may support the implementation of choice, including mutual inhibition in recurrent neural networks and the hierarchical organization of timescales for information processing across the cortex. This account also suggests that certain correlates of value are emergent rather than represented explicitly in the brain.

Response repetition biases in human perceptual decisions are explained by activity decay in competitive attractor models

Animals and humans have a tendency to repeat recent choices, a phenomenon known as choice hysteresis. The mechanism for this choice bias remains unclear. Using an established, biophysically informed model of a competitive attractor network for decision making, we found that decaying tail activity from the previous trial caused choice hysteresis, especially during difficult trials, and accurately predicted human perceptual choices. In the model, choice variability could be directionally altered through amplification or dampening of post-trial activity decay through simulated depolarizing or hyperpolarizing network stimulation. An analogous intervention using transcranial direct current stimulation (tDCS) over left dorsolateral prefrontal cortex (dlPFC) yielded a close match between model predictions and experimental results: net soma depolarizing currents increased choice hysteresis, while hyperpolarizing currents suppressed it. Residual activity in competitive attractor networks within dlPFC may thus give rise to biases in perceptual choices, which can be directionally controlled through non-invasive brain stimulation.

Autocorrelation structure at rest predicts value correlates of single neurons during reward-guided choice

Correlates of value are routinely observed in the prefrontal cortex (PFC) during reward-guided decision making. In previous work (Hunt et al., 2015), we argued that PFC correlates of chosen value are a consequence of varying rates of a dynamical evidence accumulation process. Yet within PFC, there is substantial variability in chosen value correlates across individual neurons. Here we show that this variability is explained by neurons having different temporal receptive fields of integration, indexed by examining neuronal spike rate autocorrelation structure whilst at rest. We find that neurons with protracted resting temporal receptive fields exhibit stronger chosen value correlates during choice. Within orbitofrontal cortex, these neurons also sustain coding of chosen value from choice through the delivery of reward, providing a potential neural mechanism for maintaining predictions and updating stored values during learning. These findings reveal that within PFC, variability in temporal specialisation across neurons predicts involvement in specific decision-making computations.

DOI:http://dx.doi.org/10.7554/eLife.18937.001

Decoding subjective decisions from orbitofrontal cortex

When making a subjective choice, the brain must compute a value for each option and compare those values to make a decision. The orbitofrontal cortex (OFC) is critically involved in this process, but the neural mechanisms remain obscure, in part due to limitations in our ability to measure and control the internal deliberations that can alter the dynamics of the decision process. Here, we tracked the dynamics by recovering temporally precise neural states from multi-dimensional data in OFC. During individual choices, OFC alternated between states associated with the value of two available options, with dynamics that predicted whether a subject would decide quickly or vacillate between the two alternatives. Ensembles of value-encoding neurons contributed to these states, with individual neurons shifting activity patterns as the network evaluated each option. Thus, the mechanism of subjective decision-making involves the dynamic activation of OFC states associated with each choice alternative.

Neuronal remapping and circuit persistence in economic decisions

The orbitofrontal cortex plays a central role in good-based economic decisions. When subjects make choices, neurons in this region represent the identities and values of offered and chosen goods. Notably, choices in different behavioral contexts may involve a potentially infinite variety of goods. Thus a fundamental question concerns the stability versus flexibility of the decision circuit. Here we show in rhesus monkeys that neurons encoding the identity or the subjective value of particular goods in a given context 'remap' and become associated with different goods when the context changes. At the same time, the overall organization of the decision circuit and the function of individual cells remain stable across contexts. In particular, two neurons supporting the same decision in one context also support the same decision in different contexts. These results demonstrate how the same neural circuit can underlie economic decisions involving a large variety of goods.

Diffusion Decision Model: Current Issues and History

There is growing interest in diffusion models to represent the cognitive and neural processes of speeded decision making. Sequential-sampling models like the diffusion model have a long history in psychology. They view decision making as a process of noisy accumulation of evidence from a stimulus. The standard model assumes that evidence accumulates at a constant rate during the second or two it takes to make a decision. This process can be linked to the behaviors of populations of neurons and to theories of optimality. Diffusion models have been used successfully in a range of cognitive tasks and as psychometric tools in clinical research to examine individual differences. In this review, we relate the models to both earlier and more recent research in psychology.

Capturing the temporal evolution of choice across prefrontal cortex

Activity in prefrontal cortex (PFC) has been richly described using economic models of choice. Yet such descriptions fail to capture the dynamics of decision formation. Describing dynamic neural processes has proven challenging due to the problem of indexing the internal state of PFC and its trial-by-trial variation. Using primate neurophysiology and human magnetoencephalography, we here recover a single-trial index of PFC internal states from multiple simultaneously recorded PFC subregions. This index can explain the origins of neural representations of economic variables in PFC. It describes the relationship between neural dynamics and behaviour in both human and monkey PFC, directly bridging between human neuroimaging data and underlying neuronal activity. Moreover, it reveals a functionally dissociable interaction between orbitofrontal cortex, anterior cingulate cortex and dorsolateral PFC in guiding cost-benefit decisions. We cast our observations in terms of a recurrent neural network model of choice, providing formal links to mechanistic dynamical accounts of decision-making.

DOI:http://dx.doi.org/10.7554/eLife.11945.001

In 1848, a railroad worker named Phineas Gage suffered an accident that was to secure him a place in neuroscience lore. While constructing a new railway line, a mistimed explosion propelled an iron bar into the base of his skull, where it passed behind his left eye before exiting through the top of his head. Gage survived the accident, but those who knew him reported significant changes in his personality and behaviour.

Gage’s ability to make decisions was particularly impaired by his injury. Decision-making involves weighing up the costs and benefits associated with alternative courses of action. It entails looking into the future to decide whether an anticipated reward will justify the effort or expense necessary to obtain it. This process is dependent on a region of the brain called the prefrontal cortex, the area that sustained the most damage in Phineas Gage.

While many studies have shown correlations between activity in particular parts of prefrontal cortex and the outcome of decisions, little is known about how this activity evolves over time as a decision is made. To explore this process, Hunt et al. trained macaque monkeys to choose between pairs of images that were associated with specific rewards (quantities of fruit juice) and costs (either amounts of work or fixed delays).

Electrode recordings revealed changes in prefrontal activity that varied over time as the monkeys deliberated over each pair of images, choosing for example between a large reward after a long delay versus a smaller reward immediately. This activity was consistent with a mathematical model of decision-making, which also explains data from brain imaging experiments in humans. This provides an important link between human data and electrode recordings in animals.

However, some of the patterns of activity observed in both macaques and humans appeared to reflect the speed at which decisions were made, rather than the outcome of the decisions themselves. By extracting information about decision speed on each decision from each region, it was shown that communication between regions of prefrontal cortex changes when choices are between two different amounts of work, as opposed to two different delays. Further experiments are needed to explore this phenomenon and to determine how other brain regions interact with the prefrontal cortex to support the decision-making process.

DOI:http://dx.doi.org/10.7554/eLife.11945.002