Choosing the greater of two goods: neural currencies for valuation and decision making

Inactivation of Parietal Reach Region Affects Reaching But Not Saccade Choices in Internally Guided Decisions

The posterior parietal cortex (PPC) has traditionally been considered important for awareness, spatial perception, and attention. However, recent findings provide evidence that the PPC also encodes information important for making decisions. These findings have initiated a running argument of whether the PPC is critically involved in decision making. To examine this issue, we reversibly inactivated the parietal reach region (PRR), the area of the PPC that is specialized for reaching movements, while two monkeys performed a memory-guided reaching or saccade task. The task included choices between two equally rewarded targets presented simultaneously in opposite visual fields. Free-choice trials were interleaved with instructed trials, in which a single cue presented in the peripheral visual field defined the reach and saccade target unequivocally. We found that PRR inactivation led to a strong reduction of contralesional choices, but only for reaches. On the other hand, saccade choices were not affected by PRR inactivation. Importantly, reaching and saccade movements to single instructed targets remained largely intact. These results cannot be explained as an effector-nonspecific deficit in spatial attention or awareness, since the temporary "lesion" had an impact only on reach choices. Hence, the PPR is a part of a network for reach decisions and not just reach planning.

SIGNIFICANCE STATEMENT

There has been an ongoing debate on whether the posterior parietal cortex (PPC) represents only spatial awareness, perception, and attention or whether it is also involved in decision making for actions. In this study we explore whether the parietal reach region (PRR), the region of the PPC that is specialized for reaches, is involved in the decision process. We inactivated the PRR while two monkeys performed reach and saccade choices between two targets presented simultaneously in both hemifields. We found that inactivation affected only the reach choices, while leaving saccade choices intact. These results cannot be explained as a deficit in attention, since the temporary lesion affected only the reach choices. Thus, PRR is a part of a network for making reach decisions.

Dynamics of individual perceptual decisions

Perceptual decision making is fundamental to a broad range of fields including neurophysiology, economics, medicine, advertising, law, etc. Although recent findings have yielded major advances in our understanding of perceptual decision making, decision making as a function of time and frequency (i.e., decision-making dynamics) is not well understood. To limit the review length, we focus most of this review on human findings. Animal findings, which are extensively reviewed elsewhere, are included when beneficial or necessary. We attempt to put these various findings and data sets, which can appear to be unrelated in the absence of a formal dynamic analysis, into context using published models. Specifically, by adding appropriate dynamic mechanisms (e.g., high-pass filters) to existing models, it appears that a number of otherwise seemingly disparate findings from the literature might be explained. One hypothesis that arises through this dynamic analysis is that decision making includes phasic (high pass) neural mechanisms, an evidence accumulator and/or some sort of midtrial decision-making mechanism (e.g., peak detector and/or decision boundary).

The Neuroscience of Consumer Choice

We review progress and challenges relating to scientific and applied goals of the nascent field of consumer neuroscience. Scientifically, substantial progress has been made in understanding the neurobiology of choice processes. Further advances, however, require researchers to begin clarifying the set of developmental and cognitive processes that shape and constrain choices. First, despite the centrality of preferences in theories of consumer choice, we still know little about where preferences come from and the underlying developmental processes. Second, the role of attention and memory processes in consumer choice remains poorly understood, despite importance ascribed to them in interpreting data from the field. The applied goal of consumer neuroscience concerns our ability to translate this understanding to augment prediction at the population level. Although the use of neuroscientific data for market-level predictions remains speculative, there is growing evidence of superiority in specific cases over existing market research techniques.

Resolving the neural circuits of anxiety

Although anxiety disorders represent a major societal problem demanding new therapeutic targets, these efforts have languished in the absence of a mechanistic understanding of this subjective emotional state. While it is impossible to know with certainty the subjective experience of a rodent, rodent models hold promise in dissecting well-conserved limbic circuits. The application of modern approaches in neuroscience has already begun to unmask the neural circuit intricacies underlying anxiety by allowing direct examination of hypotheses drawn from existing psychological concepts. This information points toward an updated conceptual model for what neural circuit perturbations could give rise to pathological anxiety and thereby provides a roadmap for future therapeutic development.

Dynamic Integration of Value Information into a Common Probability Currency as a Theory for Flexible Decision Making

Decisions involve two fundamental problems, selecting goals and generating actions to pursue those goals. While simple decisions involve choosing a goal and pursuing it, humans evolved to survive in hostile dynamic environments where goal availability and value can change with time and previous actions, entangling goal decisions with action selection. Recent studies suggest the brain generates concurrent action-plans for competing goals, using online information to bias the competition until a single goal is pursued. This creates a challenging problem of integrating information across diverse types, including both the dynamic value of the goal and the costs of action. We model the computations underlying dynamic decision-making with disparate value types, using the probability of getting the highest pay-off with the least effort as a common currency that supports goal competition. This framework predicts many aspects of decision behavior that have eluded a common explanation.

An integrative role for the superior colliculus in selecting targets for movements

A fundamental goal of systems neuroscience is to understand the neural mechanisms underlying decision making. The midbrain superior colliculus (SC) is known to be central to the selection of one among many potential spatial targets for movements, which represents an important form of decision making that is tractable to rigorous experimental investigation. In this review, we first discuss data from mammalian models-including primates, cats, and rodents-that inform our understanding of how neural activity in the SC underlies the selection of targets for movements. We then examine the anatomy and physiology of inputs to the SC from three key regions that are themselves implicated in motor decisions-the basal ganglia, parabrachial region, and neocortex-and discuss how they may influence SC activity related to target selection. Finally, we discuss the potential for methodological advances to further our understanding of the neural bases of target selection. Our overarching goal is to synthesize what is known about how the SC and its inputs act together to mediate the selection of targets for movements, to highlight open questions about this process, and to spur future studies addressing these questions.

Decision making from economic and signal detection perspectives: development of an integrated framework

Behavior is comprised of decisions made from moment to moment (i.e., to respond one way or another). Often, the decision maker cannot be certain of the value to be accrued from the decision (i.e., the outcome value). Decisions made under outcome value uncertainty form the basis of the economic framework of decision making. Behavior is also based on perception-perception of the external physical world and of the internal bodily milieu, which both provide cues that guide decision making. These perceptual signals are also often uncertain: another person's scowling facial expression may indicate threat or intense concentration, alternatives that require different responses from the perceiver. Decisions made under perceptual uncertainty form the basis of the signals framework of decision making. Traditional behavioral economic approaches to decision making focus on the uncertainty that comes from variability in possible outcome values, and typically ignore the influence of perceptual uncertainty. Conversely, traditional signal detection approaches to decision making focus on the uncertainty that arises from variability in perceptual signals and typically ignore the influence of outcome value uncertainty. Here, we compare and contrast the economic and signals frameworks that guide research in decision making, with the aim of promoting their integration. We show that an integrated framework can expand our ability to understand a wider variety of decision-making behaviors, in particular the complexly determined real-world decisions we all make every day.

Signatures of Value Comparison in Ventral Striatum Neurons

The ventral striatum (VS), like its cortical afferents, is closely associated with processing of rewards, but the relative contributions of striatal and cortical reward systems remains unclear. Most theories posit distinct roles for these structures, despite their similarities. We compared responses of VS neurons to those of ventromedial prefrontal cortex (vmPFC) Area 14 neurons, recorded in a risky choice task. Five major response patterns observed in vmPFC were also observed in VS: (1) offer value encoding, (2) value difference encoding, (3) preferential encoding of chosen relative to unchosen value, (4) a correlation between residual variance in responses and choices, and (5) prominent encoding of outcomes. We did observe some differences as well; in particular, preferential encoding of the chosen option was stronger and started earlier in VS than in vmPFC. Nonetheless, the close match between vmPFC and VS suggests that cortex and its striatal targets make overlapping contributions to economic choice.

A study of single neurons in the ventral striatum reveals signatures of value comparison and selection during a risky choice task, suggesting that the cortex and its striatal targets make overlapping contributions to the choice process. Read the accompanying Primer.

The neural calculations underlying reward-based choice are closely associated with a network of brain areas including the ventral striatum (VS) and ventromedial prefrontal cortex (vmPFC). Most theories ascribe distinct roles to these two structures during choice, but these differences have yet to be confirmed at the level of single neurons. We compared responses of VS neurons to those of vmPFC neurons recorded in rhesus macaques choosing between potential gambles for water rewards. We found widespread similarities in the way that VS and vmPFC neurons fire during the choice process. Neurons in both areas encoded the value of the offered gamble, the difference in value between offered gambles, and the gamble outcome. Additionally, both areas showed stronger coding for the chosen gamble than for the unchosen one and predicted choice even when we controlled for offer value. Interestingly, preferential encoding of the chosen option was stronger and started earlier in VS than in vmPFC. Nonetheless, similarities between vmPFC and VS suggest that cortex and its striatal targets make overlapping contributions to reward-based choice.

Vacillation, indecision and hesitation in moment-by-moment decoding of monkey motor cortex

When choosing actions, we can act decisively, vacillate, or suffer momentary indecision. Studying how individual decisions unfold requires moment-by-moment readouts of brain state. Here we provide such a view from dorsal premotor and primary motor cortex. Two monkeys performed a novel decision task while we recorded from many neurons simultaneously. We found that a decoder trained using 'forced choices' (one target viable) was highly reliable when applied to 'free choices'. However, during free choices internal events formed three categories. Typically, neural activity was consistent with rapid, unwavering choices. Sometimes, though, we observed presumed 'changes of mind': the neural state initially reflected one choice before changing to reflect the final choice. Finally, we observed momentary 'indecision': delay forming any clear motor plan. Further, moments of neural indecision accompanied moments of behavioral indecision. Together, these results reveal the rich and diverse set of internal events long suspected to occur during free choice.

Decision-making impairments in Parkinson's disease as a by-product of defective cost-benefit analysis and feedback processing

Studies examining decision-making in people with Parkinson's disease (PD) show impaired performance on a variety of tasks. However, there are also demonstrations that patients with PD can make optimal decisions just like healthy age-matched controls. We propose that the reason for these mixed findings is that PD does not produce a generalized impairment of decision-making, but rather affects sub-components of this process. In this review we evaluate this hypothesis by considering the empirical evidence examining decision-making in PD. We suggest that of the various stages of the decision-making process, the most affected in PD are (1) the cost-benefit analysis stage and (2) the outcome evaluation stage. We consider the implications of this proposal for research in this area.

Prefrontal cortical recordings with biomorphic MEAs reveal complex columnar-laminar microcircuits for BCI/BMI implementation

The mammalian prefrontal cortex known as the seat of high brain functions uses a six layer distribution of minicolumnar neurons to coordinate the integration of sensory information and the selection of relevant signals for goal driven behavior. To reveal the complex functionality of these columnar microcircuits we employed simultaneous recordings with several configurations of biomorphic microelectrode arrays (MEAs) within cortical layers in adjacent minicolumns, in four nohuman primates (NHPs) performing a delayed match-to-sample (DMS) visual discrimination task. We examined: (1) the functionality of inter-laminar, and inter-columnar interactions between pairs of cells in the same or different minicolumns by use of normalized cross-correlation histograms (CCH), (2) the modulation of glutamate concentration in layer 2/3, and (3) the potential interactions within these microcircuits. The results demonstrate that neurons in both infra-granular and supra-granular layers interact through inter-laminar loops, as well as through intra-laminar to produce behavioral response signals. These results provide new insights into the manner in which prefrontal cortical microcircuitry integrates sensory stimuli used to provide behaviorally relevant signals that may be implemented in brain computer/machine interfaces (BCI/BMIs) during performance of the task.

Neural mechanisms underlying human consensus decision-making

Consensus building in a group is a hallmark of animal societies, yet little is known about its underlying computational and neural mechanisms. Here, we applied a computational framework to behavioral and fMRI data from human participants performing a consensus decision-making task with up to five other participants. We found that participants reached consensus decisions through integrating their own preferences with information about the majority group members' prior choices, as well as inferences about how much each option was stuck to by the other people. These distinct decision variables were separately encoded in distinct brain areas-the ventromedial prefrontal cortex, posterior superior temporal sulcus/temporoparietal junction, and intraparietal sulcus-and were integrated in the dorsal anterior cingulate cortex. Our findings provide support for a theoretical account in which collective decisions are made through integrating multiple types of inference about oneself, others, and environments, processed in distinct brain modules.

Local morphology predicts functional organization of experienced value signals in the human orbitofrontal cortex

Experienced value representations within the human orbitofrontal cortex (OFC) are thought to be organized through an antero-posterior gradient corresponding to secondary versus primary rewards. Whether this gradient depends upon specific morphological features within this region, which displays considerable intersubject variability, remains unknown. To test the existence of such relationships, we performed a subject-by-subject analysis of fMRI data taking into account the local morphology of each individual. We tested 38 subjects engaged in a simple incentive delay task manipulating both monetary and visual erotic rewards, focusing on reward outcome (experienced value signal). The results showed reliable and dissociable primary (erotic) and secondary (monetary) experienced value signals at specific OFC sulci locations. More specifically, experienced value signal induced by monetary reward outcome was systematically located in the rostral portion of the medial orbital sulcus. Experienced value signal related to erotic reward outcome was located more posteriorly, that is, at the intersection between the caudal portion of the medial orbital sulcus and transverse orbital sulcus. Thus, the localizations of distinct experienced value signals can be predicted from the organization of the human orbitofrontal sulci. This study provides insights into the anatomo-functional parcellation of the anteroposterior OFC gradient observed for secondary versus primary rewards because there is a direct relationship between value signals at the time of reward outcome and unique OFC sulci locations.

A biologically plausible computational theory for value integration and action selection in decisions with competing alternatives

Decision making is a vital component of human and animal behavior that involves selecting between alternative options and generating actions to implement the choices. Although decisions can be as simple as choosing a goal and then pursuing it, humans and animals usually have to make decisions in dynamic environments where the value and the availability of an option change unpredictably with time and previous actions. A predator chasing multiple prey exemplifies how goals can dynamically change and compete during ongoing actions. Classical psychological theories posit that decision making takes place within frontal areas and is a separate process from perception and action. However, recent findings argue for additional mechanisms and suggest the decisions between actions often emerge through a continuous competition within the same brain regions that plan and guide action execution. According to these findings, the sensorimotor system generates concurrent action-plans for competing goals and uses online information to bias the competition until a single goal is pursued. This information is diverse, relating to both the dynamic value of the goal and the cost of acting, creating a challenging problem in integrating information across these diverse variables in real time. We introduce a computational framework for dynamically integrating value information from disparate sources in decision tasks with competing actions. We evaluated the framework in a series of oculomotor and reaching decision tasks and found that it captures many features of choice/motor behavior, as well as its neural underpinnings that previously have eluded a common explanation.

In high-pressure situations, such as driving on a highway or flying a plane, people have limited time to select between competing options while acting. Each option is usually accompanied with reward benefits (e.g., avoid traffic) and action costs (e.g., fuel consumption) that characterize the value of the option. The value and the availability of an option can change dynamically even during ongoing actions which compounds the decision-making challenge. How the brain dynamically integrates value information from disparate sources and selects between competing options is still poorly understood. In the current study, we present a neurodynamical framework to show how a distributed brain network can solve the problem of value integration and action selection in decisions with competing alternatives. It combines dynamic neural field theory with stochastic optimal control theory, and includes circuitry for perception, expected reward, effort cost and decision-making. It provides a principled way to explain both the neural and the behavioral findings from a series of visuomotor decision tasks in human and animal studies. For instance, the model shows how the competitive interactions between populations of neurons within and between sensorimotor regions can result in “spatial-averaging” movements, and how decision-variables influence neural activity and choice behavior.

Supplementary Eye Field Encodes Confidence in Decisions Under Risk

Choices are made with varying degrees of confidence, a cognitive signal representing the subjective belief in the optimality of the choice. Confidence has been mostly studied in the context of perceptual judgments, in which choice accuracy can be measured using objective criteria. Here, we study confidence in subjective value-based decisions. We recorded in the supplementary eye field (SEF) of monkeys performing a gambling task, where they had to use subjective criteria for placing bets. We found neural signals in the SEF that explicitly represent choice confidence independent from reward expectation. This confidence signal appeared after the choice and diminished before the choice outcome. Most of this neuronal activity was negatively correlated with confidence, and was strongest in trials on which the monkey spontaneously withdrew his choice. Such confidence-related activity indicates that the SEF not only guides saccade selection, but also evaluates the likelihood that the choice was optimal. This internal evaluation influences decisions concerning the willingness to bear later costs that follow from the choice or to avoid them. More generally, our findings indicate that choice confidence is an integral component of all forms of decision-making, whether they are based on perceptual evidence or on value estimations.

Detecting Functional Connectivity During Audiovisual Integration with MEG: A Comparison of Connectivity Metrics

In typical magnetoencephalography and/or electroencephalography functional connectivity analysis, researchers select one of several methods that measure a relationship between regions to determine connectivity, such as coherence, power correlations, and others. However, it is largely unknown if some are more suited than others for various types of investigations. In this study, the authors investigate seven connectivity metrics to evaluate which, if any, are sensitive to audiovisual integration by contrasting connectivity when tracking an audiovisual object versus connectivity when tracking a visual object uncorrelated with the auditory stimulus. The authors are able to assess the metrics' performances at detecting audiovisual integration by investigating connectivity between auditory and visual areas. Critically, the authors perform their investigation on a whole-cortex all-to-all mapping, avoiding confounds introduced in seed selection. The authors find that amplitude-based connectivity measures in the beta band detect strong connections between visual and auditory areas during audiovisual integration, specifically between V4/V5 and auditory cortices in the right hemisphere. Conversely, phase-based connectivity measures in the beta band as well as phase and power measures in alpha, gamma, and theta do not show connectivity between audiovisual areas. The authors postulate that while beta power correlations detect audiovisual integration in the current experimental context, it may not always be the best measure to detect connectivity. Instead, it is likely that the brain utilizes a variety of mechanisms in neuronal communication that may produce differential types of temporal relationships.

Neural components of altruistic punishment

Altruistic punishment, which occurs when an individual incurs a cost to punish in response to unfairness or a norm violation, may play a role in perpetuating cooperation. The neural correlates underlying costly punishment have only recently begun to be explored. Here we review the current state of research on the neural basis of altruism from the perspectives of costly punishment, emphasizing the importance of characterizing elementary neural processes underlying a decision to punish. In particular, we emphasize three cognitive processes that contribute to the decision to altruistically punish in most scenarios: inequity aversion, cost-benefit calculation, and social reference frame to distinguish self from others. Overall, we argue for the importance of understanding the neural correlates of altruistic punishment with respect to the core computations necessary to achieve a decision to punish.

Dissociating movement from movement timing in the rat primary motor cortex

Neural encoding of the passage of time to produce temporally precise movements remains an open question. Neurons in several brain regions across different experimental contexts encode estimates of temporal intervals by scaling their activity in proportion to the interval duration. In motor cortex the degree to which this scaled activity relies upon afferent feedback and is guided by motor output remains unclear. Using a neural reward paradigm to dissociate neural activity from motor output before and after complete spinal transection, we show that temporally scaled activity occurs in the rat hindlimb motor cortex in the absence of motor output and after transection. Context-dependent changes in the encoding are plastic, reversible, and re-established following injury. Therefore, in the absence of motor output and despite a loss of afferent feedback, thought necessary for timed movements, the rat motor cortex displays scaled activity during a broad range of temporally demanding tasks similar to that identified in other brain regions.

Between the primate and 'reptilian' brain: Rodent models demonstrate the role of corticostriatal circuits in decision making

Decision making can be defined as the flexible integration and transformation of information from the external world into action. Recently, the development of novel genetic tools and new behavioral paradigms has made it attractive to study behavior of all kinds in rodents. By some perspectives, rodents are not an acceptable model for the study of decision making due to their simpler behavior often attributed to their less extensive cortical development when compared to non-human primates. We argue that decision making can be approached with a common framework across species. We review insights from comparative anatomy that suggest the expansion of cortical-striatal connectivity is a key development in evolutionary increases in behavioral flexibility. We briefly review studies that establish a role for corticostriatal circuits in integrative decision making. Finally, we provide an overview of a few recent, highly complementary rodent decision making studies using genetic tools, revealing with new cellular and temporal resolution how, when and where information can be integrated and compared in striatal circuits to influence choice.

Neurophysiology of Reward-Guided Behavior: Correlates Related to Predictions, Value, Motivation, Errors, Attention, and Action

Many brain areas are activated by the possibility and receipt of reward. Are all of these brain areas reporting the same information about reward? Or are these signals related to other functions that accompany reward-guided learning and decision-making? Through carefully controlled behavioral studies, it has been shown that reward-related activity can represent reward expectations related to future outcomes, errors in those expectations, motivation, and signals related to goal- and habit-driven behaviors. These dissociations have been accomplished by manipulating the predictability of positively and negatively valued events. Here, we review single neuron recordings in behaving animals that have addressed this issue. We describe data showing that several brain areas, including orbitofrontal cortex, anterior cingulate, and basolateral amygdala signal reward prediction. In addition, anterior cingulate, basolateral amygdala, and dopamine neurons also signal errors in reward prediction, but in different ways. For these areas, we will describe how unexpected manipulations of positive and negative value can dissociate signed from unsigned reward prediction errors. All of these signals feed into striatum to modify signals that motivate behavior in ventral striatum and guide responding via associative encoding in dorsolateral striatum.

Decision-making under risk and ambiguity in low-birth-weight pigs

Low birth weight (LBW) in humans is a risk factor for later cognitive, behavioural and emotional problems. In pigs, LBW is associated with higher mortality, but little is known about consequences for surviving piglets. Alteration in hypothalamic-pituitary-adrenal axis function in LBW pigs suggests altered emotionality, but no behavioural indicators have been studied. Decision-making under uncertain conditions, e.g., risk or ambiguity, is susceptible to emotional influences and may provide a means of assessing long-term effects of LBW in piglets. We tested LBW (N = 8) and normal-birth-weight (NBW; N = 8) male pigs in two decision-making tasks. For decision-making under risk, we developed a simple two-choice probabilistic task, the Pig Gambling Task (PGT), where an 'advantageous' option offered small but frequent rewards and a 'disadvantageous' option offered large but infrequent rewards. The advantageous option offered greater overall gain. For decision-making under ambiguity, we used a Judgement Bias Task (JBT) where pigs were trained to make an active response to 'positive' and 'negative' tone cues (signalling large and small rewards, respectively). Responses to ambiguous tone cues were rated as more or less optimistic. LBW pigs chose the advantageous option more often in later blocks of the PGT, and were scored as less optimistic in the JBT, than NBW pigs. Our findings demonstrate that LBW pigs have developed different behavioural strategies with respect to decision-making. We propose that this is guided by changes in emotionality in LBW piglets, and we provide behavioural evidence of increased negative affect in LBW piglets.

Stimulus similarity determines the prevalence of behavioral laterality in a visual discrimination task for mice

Animal choices depend on direct sensory information, but also on the dynamic changes in the magnitude of reward. In visual discrimination tasks, the emergence of lateral biases in the choice record from animals is often described as a behavioral artifact, because these are highly correlated with error rates affecting psychophysical measurements. Here, we hypothesized that biased choices could constitute a robust behavioral strategy to solve discrimination tasks of graded difficulty. We trained mice to swim in a two-alterative visual discrimination task with escape from water as the reward. Their prevalence of making lateral choices increased with stimulus similarity and was present in conditions of high discriminability. While lateralization occurred at the individual level, it was absent, on average, at the population level. Biased choice sequences obeyed the generalized matching law and increased task efficiency when stimulus similarity was high. A mathematical analysis revealed that strongly-biased mice used information from past rewards but not past choices to make their current choices. We also found that the amount of lateralized choices made during the first day of training predicted individual differences in the average learning behavior. This framework provides useful analysis tools to study individualized visual-learning trajectories in mice.

Attention, reward, and information seeking

Decision making is thought to be guided by the values of alternative options and involve the accumulation of evidence to an internal bound. However, in natural behavior, evidence accumulation is an active process whereby subjects decide when and which sensory stimulus to sample. These sampling decisions are naturally served by attention and rapid eye movements (saccades), but little is known about how saccades are controlled to guide future actions. Here we review evidence that was discussed at a recent symposium, which suggests that information selection involves basal ganglia and cortical mechanisms and that, across different contexts, it is guided by two central factors: the gains in reward and gains in information (uncertainty reduction) associated with sensory cues.

On the challenges and mechanisms of embodied decisions

Neurophysiological studies of decision-making have focused primarily on elucidating the mechanisms of classic economic decisions, for which the relevant variables are the values of expected outcomes and action is simply the means of reporting the selected choice. By contrast, here we focus on the particular challenges of embodied decision-making faced by animals interacting with their environment in real time. In such scenarios, the choices themselves as well as their relative costs and benefits are defined by the momentary geometry of the immediate environment and change continuously during ongoing activity. To deal with the demands of embodied activity, animals require an architecture in which the sensorimotor specification of potential actions, their valuation, selection and even execution can all take place in parallel. Here, we review behavioural and neurophysiological data supporting a proposed brain architecture for dealing with such scenarios, which we argue set the evolutionary foundation for the organization of the mammalian brain.

From genes to behavior: placing cognitive models in the context of biological pathways

Connecting neural mechanisms of behavior to their underlying molecular and genetic substrates has important scientific and clinical implications. However, despite rapid growth in our knowledge of the functions and computational properties of neural circuitry underlying behavior in a number of important domains, there has been much less progress in extending this understanding to their molecular and genetic substrates, even in an age marked by exploding availability of genomic data. Here we describe recent advances in analytical strategies that aim to overcome two important challenges associated with studying the complex relationship between genes and behavior: (i) reducing distal behavioral phenotypes to a set of molecular, physiological, and neural processes that render them closer to the actions of genetic forces, and (ii) striking a balance between the competing demands of discovery and interpretability when dealing with genomic data containing up to millions of markers. Our proposed approach involves linking, on one hand, models of neural computations and circuits hypothesized to underlie behavior, and on the other hand, the set of the genes carrying out biochemical processes related to the functioning of these neural systems. In particular, we focus on the specific example of value-based decision-making, and discuss how such a combination allows researchers to leverage existing biological knowledge at both neural and genetic levels to advance our understanding of the neurogenetic mechanisms underlying behavior.

When pain meets … pain-related choice behavior and pain perception in different goal conflict situations

Individuals in pain often face the choice between avoiding pain and pursuing other equally valued goals. However, little is known about pain-related choice behavior and pain perception in goal conflict situations. Seventy-eight healthy volunteers performed a computerized task requiring repeated choices between incompatible options, differing in their effect on probability to receive painful stimulation and money. Depending on group assignment, participants chose between increased pain probability versus decreased money probability (avoidance-avoidance conflict situation); decreased pain probability versus increased money probability (approach-approach conflict situation); or decrease versus increase in both probabilities (double approach/avoidance conflict situation). During the choice task, participants rated painfulness, unpleasantness, threat, and fearfulness associated with the painful stimulation and how they felt. Longer choice latency and more choice switching were associated with higher retrospective ratings of conflict and of decision difficulty, and more equal importance placed on pain avoidance and earning money. Groups did not differ in choice behavior, pain stimulus ratings, or affect. Across groups, longer choice latencies were nonsignificantly associated with higher pain, unpleasantness, threat, and fearfulness. In the avoidance-avoidance group, more choice switching was associated with higher pain-related threat and fearfulness, and with more negative affect. These results of this study suggest that associations between choice behaviors, pain perception, and affect depend on conflict situation.

PERSPECTIVE

We present a first experimental demonstration of the relationship between pain-related choice behaviors, pain, and affect in different goal conflict situations. This experimental approach allows us to examine these relationships in a controlled fashion. Better understanding of pain-related goal conflicts and their resolution may lead to more effective pain treatment.

Bursts and heavy tails in temporal and sequential dynamics of foraging decisions

A fundamental understanding of behavior requires predicting when and what an individual will choose. However, the actual temporal and sequential dynamics of successive choices made among multiple alternatives remain unclear. In the current study, we tested the hypothesis that there is a general bursting property in both the timing and sequential patterns of foraging decisions. We conducted a foraging experiment in which rats chose among four different foods over a continuous two-week time period. Regarding when choices were made, we found bursts of rapidly occurring actions, separated by time-varying inactive periods, partially based on a circadian rhythm. Regarding what was chosen, we found sequential dynamics in affective choices characterized by two key features: (a) a highly biased choice distribution; and (b) preferential attachment, in which the animals were more likely to choose what they had previously chosen. To capture the temporal dynamics, we propose a dual-state model consisting of active and inactive states. We also introduce a satiation-attainment process for bursty activity, and a non-homogeneous Poisson process for longer inactivity between bursts. For the sequential dynamics, we propose a dual-control model consisting of goal-directed and habit systems, based on outcome valuation and choice history, respectively. This study provides insights into how the bursty nature of behavior emerges from the interaction of different underlying systems, leading to heavy tails in the distribution of behavior over time and choices.

To understand spontaneous animal behavior, two key elements must be explained: when an action is made and what is chosen. Here, we conducted a foraging experiment in which rats chose among four different foods over a continuous two-week time period. With respect to when, we found bursts of rapidly occurring responses separated by long inactive periods. With respect to what, we found biased choice behavior toward the favorite items as well as repetitive behavior, reflecting goal-directed and habitual responding, respectively. We account for the when and what components with two distinct computational mechanisms, each composed of two processes: (a) active and inactive states for the temporal dynamics, and (b) goal-directed and habitual control for the sequential dynamics. This study provides behavioral and computational insights into the dynamical properties of decision-making that determine both when an animal will act and what the animal will choose. Our findings provide an integrated framework for describing the temporal and sequential structure of everyday choices among, for example, food, music, books, brands, web-browsing and social interaction.

Decision making: the neuroethological turn

Neuroeconomics applies models from economics and psychology to inform neurobiological studies of choice. This approach has revealed neural signatures of concepts like value, risk, and ambiguity, which are known to influence decision making. Such observations have led theorists to hypothesize a single, unified decision process that mediates choice behavior via a common neural currency for outcomes like food, money, or social praise. In parallel, recent neuroethological studies of decision making have focused on natural behaviors like foraging, mate choice, and social interactions. These decisions strongly impact evolutionary fitness and thus are likely to have played a key role in shaping the neural circuits that mediate decision making. This approach has revealed a suite of computational motifs that appear to be shared across a wide variety of organisms. We argue that the existence of deep homologies in the neural circuits mediating choice may have profound implications for understanding human decision making in health and disease.

Microstimulation of the human substantia nigra alters reinforcement learning

Animal studies have shown that substantia nigra (SN) dopaminergic (DA) neurons strengthen action-reward associations during reinforcement learning, but their role in human learning is not known. Here, we applied microstimulation in the SN of 11 patients undergoing deep brain stimulation surgery for the treatment of Parkinson's disease as they performed a two-alternative probability learning task in which rewards were contingent on stimuli, rather than actions. Subjects demonstrated decreased learning from reward trials that were accompanied by phasic SN microstimulation compared with reward trials without stimulation. Subjects who showed large decreases in learning also showed an increased bias toward repeating actions after stimulation trials; therefore, stimulation may have decreased learning by strengthening action-reward associations rather than stimulus-reward associations. Our findings build on previous studies implicating SN DA neurons in preferentially strengthening action-reward associations during reinforcement learning.

Modeling task control of eye movements

In natural behavior, visual information is actively sampled from the environment by a sequence of gaze changes. The timing and choice of gaze targets, and the accompanying attentional shifts, are intimately linked with ongoing behavior. Nonetheless, modeling of the deployment of these fixations has been very difficult because they depend on characterizing the underlying task structure. Recently, advances in eye tracking during natural vision, together with the development of probabilistic modeling techniques, have provided insight into how the cognitive agenda might be included in the specification of fixations. These techniques take advantage of the decomposition of complex behaviors into modular components. A particular subset of these models casts the role of fixation as that of providing task-relevant information that is rewarding to the agent, with fixation being selected on the basis of expected reward and uncertainty about environmental state. We review this work here and describe how specific examples can reveal general principles in gaze control.

Rapid prediction of biomechanical costs during action decisions

When given a choice between actions that yield the same reward, we tend to prefer the one that requires the least effort. Recent studies have shown that humans are remarkably accurate at evaluating the effort of potential reaching actions and can predict the subtle energetic demand caused by the nonisotropic biomechanical properties of the arm. In the present study, we investigated the time course over which such information is computed and comes to influence decisions. Two independent approaches were used. First, subjects performed a reach decision task in which the time interval for deciding between two candidate reaching actions was varied from 200 to 800 ms. Second, we measured motor-evoked potential (MEPs) to single-pulse transcranial magnetic stimulation (TMS) over the primary motor cortex (M1) to probe the evolving decision at different times after stimulus presentation. Both studies yielded a consistent conclusion: that a prediction of the effort associated with candidate movements is computed very quickly and influences decisions within 200 ms after presentation of the candidate actions. Furthermore, whereas the MEPs measured 150 ms after stimulus presentation were well correlated with the choices that subjects ultimately made, later in the trial the MEP amplitudes were primarily related to the muscular requirements of the chosen movement. This suggests that corticospinal excitability (CSE) initially reflects a competition between candidate actions and later changes to reflect the processes of preparing to implement the winning action choice.

Reward value comparison via mutual inhibition in ventromedial prefrontal cortex

Recent theories suggest that reward-based choice reflects competition between value signals in the ventromedial prefrontal cortex (vmPFC). We tested this idea by recording vmPFC neurons while macaques performed a gambling task with asynchronous offer presentation. We found that neuronal activity shows four patterns consistent with selection via mutual inhibition: (1) correlated tuning for probability and reward size, suggesting that vmPFC carries an integrated value signal; (2) anti-correlated tuning curves for the two options, suggesting mutual inhibition; (3) neurons rapidly come to signal the value of the chosen offer, suggesting the circuit serves to produce a choice; and (4) after regressing out the effects of option values, firing rates still could predict choice-a choice probability signal. In addition, neurons signaled gamble outcomes, suggesting that vmPFC contributes to both monitoring and choice processes. These data suggest a possible mechanism for reward-based choice and endorse the centrality of vmPFC in that process.

Pharmacological enhancement of memory or cognition in normal subjects

The possibility of expanding memory or cognitive capabilities above the levels in high functioning individuals is a topic of intense discussion among scientists and in society at large. The majority of animal studies use behavioral endpoint measures; this has produced valuable information but limited predictability for human outcomes. Accordingly, several groups are pursuing a complementary strategy with treatments targeting synaptic events associated with memory encoding or forebrain network operations. Transcription and translation figure prominently in substrate work directed at enhancement. Notably, the question of why new proteins would be needed for a now-forming memory given that learning-driven synthesis presumably occurred throughout the immediate past has been largely ignored. Despite this conceptual problem, and some controversy, recent studies have reinvigorated the idea that selective gene manipulation is a plausible route to enhancement. Efforts to improve memory by facilitating synaptic encoding of information have also progressed, in part due of breakthroughs on mechanisms that stabilize learning-related, long-term potentiation (LTP). These advances point to a reductionistic hypothesis for a diversity of experimental results on enhancement, and identify under-explored possibilities. Cognitive enhancement remains an elusive goal, in part due to the difficulty of defining the target. The popular view of cognition as a collection of definable computations seems to miss the fluid, integrative process experienced by high functioning individuals. The neurobiological approach obviates these psychological issues to directly test the consequences of improving throughput in networks underlying higher order behaviors. The few relevant studies testing drugs that selectively promote excitatory transmission indicate that it is possible to expand cortical networks engaged by complex tasks and that this is accompanied by capabilities not found in normal animals.

A common currency for the computation of motivational values in the human striatum

Reward comparison in the brain is thought to be achieved through the use of a 'common currency', implying that reward value representations are computed on a unique scale in the same brain regions regardless of the reward type. Although such a mechanism has been identified in the ventro-medial prefrontal cortex and ventral striatum in the context of decision-making, it is less clear whether it similarly applies to non-choice situations. To answer this question, we scanned 38 participants with fMRI while they were presented with single cues predicting either monetary or erotic rewards, without the need to make a decision. The ventral striatum was the main brain structure to respond to both cues while showing increasing activity with increasing expected reward intensity. Most importantly, the relative response of the striatum to monetary vs erotic cues was correlated with the relative motivational value of these rewards as inferred from reaction times. Similar correlations were observed in a fronto-parietal network known to be involved in attentional focus and motor readiness. Together, our results suggest that striatal reward value signals not only obey to a common currency mechanism in the absence of choice but may also serve as an input to adjust motivated behaviour accordingly.

Motor effort alters changes of mind in sensorimotor decision making

After committing to an action, a decision-maker can change their mind to revise the action. Such changes of mind can even occur when the stream of information that led to the action is curtailed at movement onset. This is explained by the time delays in sensory processing and motor planning which lead to a component at the end of the sensory stream that can only be processed after initiation. Such post-initiation processing can explain the pattern of changes of mind by asserting an accumulation of additional evidence to a criterion level, termed change-of-mind bound. Here we test the hypothesis that physical effort associated with the movement required to change one's mind affects the level of the change-of-mind bound and the time for post-initiation deliberation. We varied the effort required to change from one choice target to another in a reaching movement by varying the geometry of the choice targets or by applying a force field between the targets. We show that there is a reduction in the frequency of change of mind when the separation of the choice targets would require a larger excursion of the hand from the initial to the opposite choice. The reduction is best explained by an increase in the evidence required for changes of mind and a reduced time period of integration after the initial decision. Thus the criteria to revise an initial choice is sensitive to energetic costs.

Bayesian deterministic decision making: a normative account of the operant matching law and heavy-tailed reward history dependency of choices

The decision making behaviors of humans and animals adapt and then satisfy an “operant matching law” in certain type of tasks. This was first pointed out by Herrnstein in his foraging experiments on pigeons. The matching law has been one landmark for elucidating the underlying processes of decision making and its learning in the brain. An interesting question is whether decisions are made deterministically or probabilistically. Conventional learning models of the matching law are based on the latter idea; they assume that subjects learn choice probabilities of respective alternatives and decide stochastically with the probabilities. However, it is unknown whether the matching law can be accounted for by a deterministic strategy or not. To answer this question, we propose several deterministic Bayesian decision making models that have certain incorrect beliefs about an environment. We claim that a simple model produces behavior satisfying the matching law in static settings of a foraging task but not in dynamic settings. We found that the model that has a belief that the environment is volatile works well in the dynamic foraging task and exhibits undermatching, which is a slight deviation from the matching law observed in many experiments. This model also demonstrates the double-exponential reward history dependency of a choice and a heavier-tailed run-length distribution, as has recently been reported in experiments on monkeys.

Hippocampal-cortical interaction in decision making

When making a decision it is often necessary to consider the available alternatives in order to choose the most appropriate option. This deliberative process, where the pros and cons of each option are considered, relies on memories of past actions and outcomes. The hippocampus and prefrontal cortex are required for memory encoding, memory retrieval and decision making, but it is unclear how these areas support deliberation. Here we examine the potential neural substrates of these processes in the rat. The rat is a powerful model to investigate the network mechanisms underlying deliberation in the mammalian brain given the anatomical and functional conservation of its hippocampus and prefrontal cortex to other mammalian systems. Importantly, it is amenable to large scale neural recording while performing laboratory tasks that exploit its natural decision-making behavior. Focusing on findings in the rat, we discuss how hippocampal-cortical interactions could provide a neural substrate for deliberative decision making.

Neural correlates of temporal credit assignment in the parietal lobe

Empirical studies of decision making have typically assumed that value learning is governed by time, such that a reward prediction error arising at a specific time triggers temporally-discounted learning for all preceding actions. However, in natural behavior, goals must be acquired through multiple actions, and each action can have different significance for the final outcome. As is recognized in computational research, carrying out multi-step actions requires the use of credit assignment mechanisms that focus learning on specific steps, but little is known about the neural correlates of these mechanisms. To investigate this question we recorded neurons in the monkey lateral intraparietal area (LIP) during a serial decision task where two consecutive eye movement decisions led to a final reward. The underlying decision trees were structured such that the two decisions had different relationships with the final reward, and the optimal strategy was to learn based on the final reward at one of the steps (the “F” step) but ignore changes in this reward at the remaining step (the “I” step). In two distinct contexts, the F step was either the first or the second in the sequence, controlling for effects of temporal discounting. We show that LIP neurons had the strongest value learning and strongest post-decision responses during the transition after the F step regardless of the serial position of this step. Thus, the neurons encode correlates of temporal credit assignment mechanisms that allocate learning to specific steps independently of temporal discounting.

Nonhuman gamblers: lessons from rodents, primates, and robots

The search for neuronal and psychological underpinnings of pathological gambling in humans would benefit from investigating related phenomena also outside of our species. In this paper, we present a survey of studies in three widely different populations of agents, namely rodents, non-human primates, and robots. Each of these populations offers valuable and complementary insights on the topic, as the literature demonstrates. In addition, we highlight the deep and complex connections between relevant results across these different areas of research (i.e., cognitive and computational neuroscience, neuroethology, cognitive primatology, neuropsychiatry, evolutionary robotics), to make the case for a greater degree of methodological integration in future studies on pathological gambling.

Neuronal origins of choice variability in economic decisions

To investigate the mechanisms through which economic decisions are formed, I examined the activity of neurons in the orbitofrontal cortex while monkeys chose between different juice types. Different classes of cells encoded the value of individual offers (offer value), the value of the chosen option (chosen value), or the identity of the chosen juice (chosen juice). Choice variability was partly explained by the tendency to repeat choices (choice hysteresis). Surprisingly, near-indifference decisions did not reflect fluctuations in the activity of offer value cells. In contrast, near-indifference decisions correlated with fluctuations in the preoffer activity of chosen juice cells. After the offer, the activity of chosen juice cells reflected the decision difficulty but did not resemble a race-to-threshold. Finally, chosen value cells presented an "activity overshooting" closely related to the decision difficulty and possibly due to fluctuations in the relative value of the juices. This overshooting was independent of choice hysteresis.

A neural mechanism underlying failure of optimal choice with multiple alternatives

Despite widespread interest in neural mechanisms of decision-making, most investigations focus on decisions between just two options. Here we adapt a biophysically plausible model of decision-making to predict how a key decision variable, the value difference signal-encoding how much better one choice is than another-changes with the value of a third, but unavailable, alternative. The model predicts a surprising failure of optimal decision-making: greater difficulty choosing between two options in the presence of a third very poor, as opposed to very good, alternative. Both investigation of human decision-making and functional magnetic resonance imaging-based measurements of value difference signals in ventromedial prefrontal cortex (vmPFC) bore out this prediction. The vmPFC signal decreased in the presence of low-value third alternatives, and vmPFC effect sizes predicted individual variation in suboptimal decision-making in the presence of multiple alternatives. The effect contrasts with that of divisive normalization in parietal cortex.

The reward value of sucrose in leptin-deficient obese mice

Leptin-deficient patients report higher "liking" ratings for food, and leptin replacement therapy normalizes these ratings even before weight loss is achieved. Since animals cannot report their ratings, we studied the relationship between leptin and food reward in leptin-deficient ob/ob mice using a optogenetic assay that quantifies the reward value of sucrose. In this assay, mice chose between one sipper dispensing the artificial sweetener sucralose coupled to optogenetic activation of dopaminergic (DA) neurons, and another sipper dispensing sucrose. We found that the reward value of sucrose was high under a state of leptin deficiency, as well as at a dose of leptin that does not suppress food intake (12.5 ng/h). Treatment with higher doses of leptin decreased the reward value of sucrose before weight loss was achieved (100 ng/h), as seen in leptin-deficient patients. These results phenocopy in mice the behavior of leptin-deficient patients.

Decision making as a window on cognition

A decision is a commitment to a proposition or plan of action based on information and values associated with the possible outcomes. The process operates in a flexible timeframe that is free from the immediacy of evidence acquisition and the real time demands of action itself. Thus, it involves deliberation, planning, and strategizing. This Perspective focuses on perceptual decision making in nonhuman primates and the discovery of neural mechanisms that support accuracy, speed, and confidence in a decision. We suggest that these mechanisms expose principles of cognitive function in general, and we speculate about the challenges and directions before the field.

Understanding decision neuroscience: a multidisciplinary perspective and neural substrates

The neuroscience of decision making is a rapidly evolving multidisciplinary research area that employs neuroscientific techniques to explain various parameters associated with decision making behavior. In this chapter, we emphasize the role of multiple disciplines such as psychology, economics, neuroscience, and computational approaches in understanding the phenomenon of decision making. Further, we present a theoretical approach that suggests understanding the building blocks of decision making as bottom-up processes and integrate these with top-down modulatory factors. Relevant neurophysiological and neuroimaging findings that have used the building-block approach are reviewed. A unifying framework emphasizing multidisciplinary views would bring further insights into the active research area of decision making. Pointing to future directions for research, we focus on the role of computational approaches in such a unifying framework.

Too good to be true: rhesus monkeys react negatively to better-than-expected offers

To succeed in a dynamically changing world, animals need to predict their environments. Humans, in fact, exhibit such a strong desire for consistency that one of the most well-established findings in social psychology is the effort people make to maintain consistency among their beliefs, attitudes, and behavior. However, displeasure with unpredictability leads to a potential paradox, because a positive outcome that exceeds one's expectations often leads to increased subjective value and positive affect, not the opposite. We tested the hypothesis that two evolutionarily-conserved evaluation processes underlie goal-directed behavior: (1) consistency, concerned with prediction errors, and (2) valuation, concerned with outcome utility. Rhesus monkeys (Macaca mulatta) viewed a food item and then were offered an identical, better, or worse food, which they could accept or reject. The monkeys ultimately accepted all offers, attesting to the influence of the valuation process. However, they were slower to accept the unexpected offers, and they exhibited aversive reactions, especially to the better-than-expected offers, repeatedly turning their heads and looking away before accepting the food item. Our findings (a) provide evidence for two separable evaluation processes in primates, consistency and value assessment, (b) reveal a direct relationship between consistency assessment and emotional processes, and (c) show that our wariness with events that are much better than expected is shared with other social primates.