Choosing the greater of two goods: neural currencies for valuation and decision making

Conceptual representations in goal-directed decision making

Emerging evidence suggests that the long-established distinction between habit-based and goal-directed decision-making mechanisms can also be sustained in humans. Although the habit-based system has been extensively studied in humans, the goal-directed system is less well characterized. This review brings to that task the distinction between conceptual and nonconceptual representational mechanisms. Conceptual representations are structured out of semantic constituents (concepts)--the use of which requires an ability to perform some language-like syntactic processing. Decision making--as investigated by neuroscience and psychology--is normally studied in isolation from questions about concepts as studied in philosophy and cognitive psychology. We ask what role concepts play in the "goal-directed" decision-making system. We argue that one fruitful way of studying this system in humans is to investigate the extent to which it deploys conceptual representations.

Multimodal activity in the parietal cortex

Goal-directed behavior can be thought of as dynamic links between sensory stimuli and motor acts. Neural correlates of many of the intermediate events of both auditory and visual goal-directed behaviors are found in the posterior parietal cortex. Here, we review studies that have focused on how neurons in the lateral intraparietal area (area LIP) differentially process auditory and visual stimuli. Together, these studies suggest that area LIP contains a modality-dependent representation that is highly dependent on behavioral context.

Local pattern classification differentiates processes of economic valuation

For effective decision making, individuals must be able to form subjective values from many types of information. Yet, the neural mechanisms that underlie potential differences in value computation across different decision scenarios are incompletely understood. Here, we used functional magnetic resonance imaging (fMRI), in conjunction with the machine learning technique of support vector machines (SVM), to identify brain regions that contain unique local information associated with different types of valuation. We used a combinatoric approach that evaluated the unique contributions of different brain regions to model generalization strength. Local voxel patterns in left posterior parietal cortex contained unique information differentiating probabilistic and intertemporal valuation, a result that was not accessible using standard fMRI analyses. We conclude that the early valuation phases for these reward types differ on a fine spatial scale, suggesting the existence of computational topographies along the value construction pathway.

Statistical decision theory to relate neurons to behavior in the study of covert visual attention

Scrutiny of the numerous physiology and imaging studies of visual attention reveal that integration of results from neuroscience with the classic theories of visual attention based on behavioral work is not simple. The different subfields have pursued different questions, used distinct experimental paradigms and developed diverse models. The purpose of this review is to use statistical decision theory and computational modeling to relate classic theories of attention in psychological research to neural observables such as mean firing rate or functional imaging BOLD response, tuning functions, Fano factor, neuronal index of detectability and area under the receiver operating characteristic (ROC). We focus on cueing experiments and attempt to distinguish two major leading theories in the study of attention: limited resources model/increased sensitivity vs. selection/differential weighting. We use Bayesian ideal observer (BIO) modeling, in which predictive cues or prior knowledge change the differential weighting (prior) of sensory information to generate predictions of behavioral and neural observables based on Gaussian response variables and Poisson process neural based models. The ideal observer model can be modified to represent a number of classic psychological theories of visual attention by including hypothesized human attentional limited resources in the same way sequential ideal observer analysis has been used to include physiological processing components of human spatial vision (Geisler, W. S. (1989). Sequential ideal-observer analysis of visual discrimination. Psychological Review 96, 267-314.). In particular we compare new biologically plausible implementations of the BIO and variant models with limited resources. We find a close relationship between the behavioral effects of cues predicted by the models developed in the field of human psychophysics and their neuron-based analogs. Critically, we show that cue effects on experimental observables such as mean neural activity, variance, Fano factor and neuronal index of detectability can be consistent with the two major theoretical models of attention depending on whether the neuron is assumed to be computing likelihoods, log-likelihoods or a simple model operating directly on the Poisson variable. Change in neuronal tuning functions can also be consistent with both theories depending on whether the change in tuning is along the dimension being experimentally cued or a different dimension. We show that a neuron's sensitivity appropriately measured using the area under the Receive Operating Characteristic curve can be used to distinguish across both theories and is robust to the many transformations of the decision variable. We provide a summary table with the hope that it might provide some guidance in interpreting past results as well as planning future studies.

A common cortical circuit mechanism for perceptual categorical discrimination and veridical judgment

Perception involves two types of decisions about the sensory world: identification of stimulus features as analog quantities, or discrimination of the same stimulus features among a set of discrete alternatives. Veridical judgment and categorical discrimination have traditionally been conceptualized as two distinct computational problems. Here, we found that these two types of decision making can be subserved by a shared cortical circuit mechanism. We used a continuous recurrent network model to simulate two monkey experiments in which subjects were required to make either a two-alternative forced choice or a veridical judgment about the direction of random-dot motion. The model network is endowed with a continuum of bell-shaped population activity patterns, each representing a possible motion direction. Slow recurrent excitation underlies accumulation of sensory evidence, and its interplay with strong recurrent inhibition leads to decision behaviors. The model reproduced the monkey's performance as well as single-neuron activity in the categorical discrimination task. Furthermore, we examined how direction identification is determined by a combination of sensory stimulation and microstimulation. Using a population-vector measure, we found that direction judgments instantiate winner-take-all (with the population vector coinciding with either the coherent motion direction or the electrically elicited motion direction) when two stimuli are far apart, or vector averaging (with the population vector falling between the two directions) when two stimuli are close to each other. Interestingly, for a broad range of intermediate angular distances between the two stimuli, the network displays a mixed strategy in the sense that direction estimates are stochastically produced by winner-take-all on some trials and by vector averaging on the other trials, a model prediction that is experimentally testable. This work thus lends support to a common neurodynamic framework for both veridical judgment and categorical discrimination in perceptual decision making.

Heisenberg's roadmap guides our journey to the small cognitive world of Drosophila

Professor Martin Heisenberg is one of the pioneers in the exploration of neuroethology. With his inspiration and earnest help, we employed the fruitfly as a model system to investigate the underlying neural mechanism of cognitive behaviors. Here, we recalled the help from Martin in the early years and introduced some findings from our lab about visual cognition behaviors in Drosophila, such as decision making, selective attention, and experience-dependent visual pattern recognition. From the results so far, the circuit composed of mushroom bodies, central complex, and dopaminergic neurons may play an essential role in these behaviors.

Temporal Dynamics Underlying Perceptual Decision Making: Insights from the Interplay between an Attractor Model and Parietal Neurophysiology

Recent neurophysiological studies in awake, behaving primates have revealed that neurons in certain brain areas appear to integrate sensory evidence over time during the performance of perceptual decision-making tasks. Neurons in the lateral intraparietal area (LIP) of rhesus monkeys exhibit such decision-related signals while the animals view and judge the direction of a visual motion display. Further investigation of this temporal integration process using brief perturbations of the sensory evidence has suggested that LIP neurons do not integrate evidence in a perfect, linear manner. We describe how a biophysically-plausible attractor network model can account for many aspects of the temporal dynamics of neural activity during these perceptual decisions. We also review a larger set of models and explain how the dynamics during and after temporal integration can help to distinguish the underlying neural mechanisms. Finally, we propose some crucial theoretically-motivated experiments that are needed to test among models.

Sensory coding of nest-site value in honeybee swarms

This study investigates the first stage of the decision-making process of a honeybee swarm as it chooses a nest site: how a scout bee codes the value of a potential nest site in the waggle dances she produces to represent this site. We presented honeybee swarms with a two-alternative choice between a high-value site and a medium-value site and recorded the behavior of individually identifiable scout bees as they reported on these two alternatives. We found that bees performed equally lengthy inspections at the two sites, but that, on the swarm cluster, they performed more dance circuits per bee for the high-value site. We also found that there was much individual-level noise in the coding of site value, but that there were clear population-level differences in total dance circuits produced for the two sites. The first bee to find a site had a high probability of reporting the site with a waggle dance, regardless of its value. This discoverer-should-dance phenomenon may help ensure that a swarm gives attention to all discovered sites. There was rapid decay in the dance response; the number of dance circuits produced by a bee after visiting a site decreased linearly over sequential visits, and eventually each bee ceased visiting her site. This decay, or `leakage', in the accumulation of bees at a site improves a swarm's decision-making ability by helping a swarm avoid making fast-decision errors.

Impact of network topology on decision-making

The dynamical behaviors of a neural system are strongly influenced by its network structure. The present study investigated how the network structure influences decision-making behaviors in the brain. We considered a recurrent network model with four different topologies, namely, regular, random, small-world and scale-free. We found that the small-world network has the best performance in decision-making for low noise, whereas the random network is most robust when noise is strong. The four networks also exhibit different behaviors in the case of neuronal damage. The performances of the regular and the small-world networks are severely degraded in distributed damage, but not in clustered damage. The random and the scale-free networks are, on the other hand, quite robust to both types of damage. Furthermore, the small-world network has the best performance in strong distributed damage.

Decision making in recurrent neuronal circuits

Decision making has recently emerged as a central theme in neurophysiological studies of cognition, and experimental and computational work has led to the proposal of a cortical circuit mechanism of elemental decision computations. This mechanism depends on slow recurrent synaptic excitation balanced by fast feedback inhibition, which not only instantiates attractor states for forming categorical choices but also long transients for gradually accumulating evidence in favor of or against alternative options. Such a circuit endowed with reward-dependent synaptic plasticity is able to produce adaptive choice behavior. While decision threshold is a core concept for reaction time tasks, it can be dissociated from a general decision rule. Moreover, perceptual decisions and value-based economic choices are described within a unified framework in which probabilistic choices result from irregular neuronal activity as well as iterative interactions of a decision maker with an uncertain environment or other unpredictable decision makers in a social group.

Gambling for Gatorade: risk-sensitive decision making for fluid rewards in humans

Determining how both humans and animals make decisions in risky situations is a central problem in economics, experimental psychology, behavioral economics, and neurobiology. Typically, humans are risk seeking for gains and risk averse for losses, while animals may display a variety of preferences under risk depending on, amongst other factors, internal state. Such differences in behavior may reflect major cognitive and cultural differences or they may reflect differences in the way risk sensitivity is probed in humans and animals. Notably, in most studies humans make one or a few choices amongst hypothetical or real monetary options, while animals make dozens of repeated choices amongst options offering primary rewards like food or drink. To address this issue, we probed risk-sensitive decision making in human participants using a paradigm modeled on animal studies, in which rewards were either small squirts of Gatorade or small amounts of real money. Possible outcomes and their probabilities were not made explicit in either case. We found that individual patterns of decision making were strikingly similar for both juice and for money, both in overall risk preferences and in trial-to-trial effects of reward outcome on choice. Comparison with decisions made by monkeys for juice in a similar task revealed highly similar gambling styles. These results unite known patterns of risk-sensitive decision making in human and nonhuman primates and suggest that factors such as the way a decision is framed or internal state may underlie observed variation in risk preferences between and within species.

From biophysics to cognition: reward-dependent adaptive choice behavior

In neurobiological studies of various cognitive abilities, neuroscientists use mathematical models to fit behavioral data from well-controlled experiments and look for neural activities that are correlated with parameters in those models. The pinpointed neural correlates are often taken as evidence that a given task is performed according to the prescription of the applied model, and the relevant brain areas encode parameters of such a model. However, to go beyond correlations toward causal understanding, it is necessary to elucidate at multiple levels the neural circuit mechanisms of cognitive processes. This review focuses on recent studies of reward-based decision-making that have begun to tackle this challenge.

Decision making, movement planning and statistical decision theory

We discuss behavioral studies directed at understanding how probability information is represented in motor and economic tasks. By formulating the behavioral tasks in the language of statistical decision theory, we can compare performance in equivalent tasks in different domains. Subjects in traditional economic decision-making tasks often misrepresent the probability of rare events and typically fail to maximize expected gain. By contrast, subjects in mathematically equivalent movement tasks often choose movement strategies that come close to maximizing expected gain. We discuss the implications of these different outcomes, noting the evident differences between the source of uncertainty and how information about uncertainty is acquired in motor and economic tasks.

Value representations in the primate striatum during matching behavior

Choosing the most valuable course of action requires knowing the outcomes associated with the available alternatives. The striatum may be important for representing the values of actions. We examined this in monkeys performing an oculomotor choice task. The activity of phasically active neurons (PANs) in the striatum covaried with two classes of information: action-values and chosen-values. Action-value PANs were correlated with value estimates for one of the available actions, and these signals were frequently observed before movement execution. Chosen-value PANs were correlated with the value of the action that had been chosen, and these signals were primarily observed later in the task, immediately before or persistently after movement execution. These populations may serve distinct functions mediated by the striatum: some PANs may participate in choice by encoding the values of the available actions, while other PANs may participate in evaluative updating by encoding the reward value of chosen actions.

Cognitive influences on risk-seeking by rhesus macaques

Humans and other animals are idiosyncratically sensitive to risk, either preferring or avoiding options having the same value but differing in uncertainty. Many explanations for risk sensitivity rely on the non-linear shape of a hypothesized utility curve. Because such models do not place any importance on uncertainty per se, utility curve-based accounts predict indifference between risky and riskless options that offer the same distribution of rewards. Here we show that monkeys strongly prefer uncertain gambles to alternating rewards with the same payoffs, demonstrating that uncertainty itself contributes to the appeal of risky options. Based on prior observations, we hypothesized that the appeal of the risky option is enhanced by the salience of the potential jackpot. To test this, we subtly manipulated payoffs in a second gambling task. We found that monkeys are more sensitive to small changes in the size of the large reward than to equivalent changes in the size of the small reward, indicating that they attend preferentially to the jackpots. Together, these results challenge utility curve-based accounts of risk sensitivity, and suggest that psychological factors, such as outcome salience and uncertainty itself, contribute to risky decision-making.

Recruitment of a head-turning synergy by low-frequency activity in the primate superior colliculus

Low-frequency activity within the oculomotor system helps bridge sensation and action. Given ocular stability, low-frequency activity sustained by some neurons within the intermediate and deep superior colliculus (dSC) is assumed to be separated from motor output. However, the dSC is an orienting structure and the influence of low-frequency dSC activity at other effectors remains untested. We studied this by simultaneously recording activity from saccade-related dSC neurons and electromyographic (EMG) activity from neck muscles that turn the head. Monkeys performed a gap-saccade paradigm with varying levels of reward expectancy. Despite head restraint and even for relatively small target eccentricities (<or=10 degrees ), increasing reward expectancy for a given target increased the level of low-frequency activity on dSC neurons encoding saccades to the rewarded target and increased the recruitment of a neck muscle synergy that would turn the head toward the target. The magnitude of neck muscle recruitment correlated positively on a trial-by-trial basis with the level of low-frequency dSC activity, and such correlations were optimized when neck muscle activity was shifted about 20 ms later to account for delays in the tecto-reticulo-spinal pathway. Further, dSC activity discriminated about the side of target presentation approximately 11 ms earlier than neck EMG activity. Considered alongside neck EMG responses evoked causally by SC stimulation, our results are consistent with low-frequency dSC activity recruiting a head-turning synergy. Our results support a brain stem circuit wherein the magnitude of neck muscle recruitment reflects the difference in comparative low-frequency activation across both dSCs, perhaps because of mutually inhibitory interactions within downstream head premotor circuits.

Reduced posterior mesofrontal cortex activation by risky rewards in substance-dependent patients

Substance-dependent individuals show disadvantageous decision-making, as well as alterated frontocortical recruitment when performing experimental tasks. We investigated whether substance-dependent patients (SDP) would show blunted recruitment of posterior mesofrontal cortex (PMC) by a conflict between concurrently increasing reward and risk of penalty in a monetary game of "chicken." SDP and controls performed: motor control (no reward) trials, guaranteed reward trials in which reward was not at risk, and risky trials where subjects were required to terminate their reward accrual before a secret varying time limit or else "bust" and forfeit that trial's winnings (low penalty) or the current trial's winnings plus an equal amount of previous winnings (high penalty). Reward accrual duration at risk of "busting" correlated negatively with trait neuroticism. The contrast between winning guaranteed reward versus non-reward activated the caudate head bilaterally in SDP but not controls. Accumulation of money at risk of low- or high-penalty (contrasted with accumulating guaranteed money) activated the PMC in both groups, but with a greater magnitude and more anterior extent in controls. Pre-decision signal increase in a PMC volume of interest negatively correlated with risk-taking in low-penalty trials, and was blunted in SDP relative to controls under both penalty conditions after controlling for individual differences in actual risk-taking and the higher neuroticism of SDP. These data suggest that SDP are characterized by a combination of: (a) striatal hypersensitivity to reward, and (b) under-recruitment of the specialized conflict-monitoring circuitry of the PMC when reward entails potential penalties.

Drosophila egg-laying site selection as a system to study simple decision-making processes

The ability to select a better option from multiple acceptable ones is important for animals to optimize their resources. The mechanisms that underlie such decision-making processes are not well understood. We found that selection of egg-laying site in Drosophila melanogaster is a suitable system to probe the neural circuit that governs simple decision-making processes. First, Drosophila females pursue active probing of the environment before depositing each egg, apparently to evaluate site quality for every egg. Second, Drosophila females can either accept or reject a sucrose-containing medium, depending on the context. Last, communication of the "acceptability" of the sucrose-containing medium as an egg-laying option to the reproductive system depends on the function of a group of insulin-like peptide 7 (ILP7)-producing neurons. These findings suggest that selection of egg-laying site involves a simple decision-making process and provide an entry point toward a systematic dissection of this process.

The diffusion decision model: theory and data for two-choice decision tasks

The diffusion decision model allows detailed explanations of behavior in two-choice discrimination tasks. In this article, the model is reviewed to show how it translates behavioral data-accuracy, mean response times, and response time distributions-into components of cognitive processing. Three experiments are used to illustrate experimental manipulations of three components: stimulus difficulty affects the quality of information on which a decision is based; instructions emphasizing either speed or accuracy affect the criterial amounts of information that a subject requires before initiating a response; and the relative proportions of the two stimuli affect biases in drift rate and starting point. The experiments also illustrate the strong constraints that ensure the model is empirically testable and potentially falsifiable. The broad range of applications of the model is also reviewed, including research in the domains of aging and neurophysiology.

The diffusion decision model: theory and data for two-choice decision tasks

The diffusion decision model allows detailed explanations of behavior in two-choice discrimination tasks. In this article, the model is reviewed to show how it translates behavioral data-accuracy, mean response times, and response time distributions-into components of cognitive processing. Three experiments are used to illustrate experimental manipulations of three components: stimulus difficulty affects the quality of information on which a decision is based; instructions emphasizing either speed or accuracy affect the criterial amounts of information that a subject requires before initiating a response; and the relative proportions of the two stimuli affect biases in drift rate and starting point. The experiments also illustrate the strong constraints that ensure the model is empirically testable and potentially falsifiable. The broad range of applications of the model is also reviewed, including research in the domains of aging and neurophysiology.

Currents contributing to decision making in neurons B31/B32 of Aplysia

Biophysical properties of neurons contributing to the ability of an animal to decide whether or not to respond were examined. B31/B32, two pairs of bilaterally symmetrical Aplysia neurons, are major participants in deciding to initiate a buccal motor program, the neural correlate of a consummatory feeding response. B31/B32 respond to an adequate stimulus after a delay, during which time additional stimuli influence the decision to respond. B31/B32 then respond with a ramp depolarization followed by a sustained soma depolarization and axon spiking that is the expression of a commitment to respond to food. Four currents contributing to decision making in B31/B32 were characterized, and their functional effects were determined, in current- and voltage-clamp experiments and with simulations. Inward currents arising from slow muscarinic transmission were characterized. These currents contribute to the B31/B32 depolarization. Their slow activation kinetics contribute to the delay preceding B31/B32 activity. After the delay, inward currents affect B31/B32 in the context of two endogenous inactivating outward currents: a delayed rectifier K+ current (I(K-V)) and an A-type K+ current (I(K-A)), as well as a high-threshold noninactivating outward current (I(maintained)). Hodgkin-Huxley kinetic analyses were performed on the outward currents. Simulations using equations from these analyses showed that I(K-V) and I(K-A) slow the ramp depolarization preceding the sustained depolarization. The three outward currents contribute to braking the B31/B32 depolarization and keeping the sustained depolarization at a constant voltage. The currents identified are sufficient to explain the properties of B31/B32 that play a role in generating the decision to feed.

Decoding of temporal intervals from cortical ensemble activity

Neurophysiological, neuroimaging, and lesion studies point to a highly distributed processing of temporal information by cortico-basal ganglia-thalamic networks. However, there are virtually no experimental data on the encoding of behavioral time by simultaneously recorded cortical ensembles. We predicted temporal intervals from the activity of hundreds of neurons recorded in motor and premotor cortex as rhesus monkeys performed self-timed hand movements. During the delay periods, when animals had to estimate temporal intervals and prepare hand movements, neuronal ensemble activity encoded both the time that elapsed from the previous hand movement and the time until the onset of the next. The neurons that were most informative of these temporal intervals increased or decreased their rates throughout the delay until reaching a threshold value, at which point a movement was initiated. Variability in the self-timed delays was explainable by the variability of neuronal rates, but not of the threshold. In addition to predicting temporal intervals, the same neuronal ensemble activity was informative for generating predictions that dissociated the delay periods of the task from the movement periods. Left hemispheric areas were the best source of predictions in one bilaterally implanted monkey overtrained to perform the task with the right hand. However, after that monkey learned to perform the task with the left hand, its left hemisphere continued and the right hemisphere started contributing to the prediction. We suggest that decoding of temporal intervals from bilaterally recorded cortical ensembles could improve the performance of neural prostheses for restoration of motor function.

Extending a biologically inspired model of choice: multi-alternatives, nonlinearity and value-based multidimensional choice

The leaky competing accumulator (LCA) is a biologically inspired model of choice. It describes the processes of leaky accumulation and competition observed in neuronal populations during choice tasks and it accounts for reaction time distributions observed in psychophysical experiments. This paper discusses recent analyses and extensions of the LCA model. First, it reviews the dynamics and examines the conditions that make the model achieve optimal performance. Second, it shows that nonlinearities of the type present in biological neurons improve performance when the number of choice alternatives increases. Third, the model is extended to value-based choice, where it is shown that nonlinearities in the value function explain risk aversion in risky choice and preference reversals in choice between alternatives characterized across multiple dimensions.

Decision theory: what "should" the nervous system do?

The purpose of our nervous system is to allow us to successfully interact with our environment. This normative idea is formalized by decision theory that defines which choices would be most beneficial. We live in an uncertain world, and each decision may have many possible outcomes; choosing the best decision is thus complicated. Bayesian decision theory formalizes these problems in the presence of uncertainty and often provides compact models that predict observed behavior. With its elegant formalization of the problems faced by the nervous system, it promises to become a major inspiration for studies in neuroscience.

Social Decision-Making: Insights from Game Theory and Neuroscience

By combining the models and tasks of Game Theory with modern psychological and neuroscientific methods, the neuroeconomic approach to the study of social decision-making has the potential to extend our knowledge of brain mechanisms involved in social decisions and to advance theoretical models of how we make decisions in a rich, interactive environment. Research has already begun to illustrate how social exchange can act directly on the brain's reward system, how affective factors play an important role in bargaining and competitive games, and how the ability to assess another's intentions is related to strategic play. These findings provide a fruitful starting point for improved models of social decision-making, informed by the formal mathematical approach of economics and constrained by known neural mechanisms.

Estimating the hidden learning representations

Successful adaptation relies on the ability to learn the consequence of our actions in different environments. However, understanding the neural bases of this ability still represents one of the great challenges of system neuroscience. In fact, the neuronal plasticity changes occurring during learning cannot be fully controlled experimentally and their evolution is hidden. Our approach is to provide hypotheses about the structure and dynamics of the hidden plasticity changes using behavioral learning theory. In fact, behavioral models of animal learning provide testable predictions about the hidden learning representations by formalizing their relation with the observables of the experiment (stimuli, actions and outcomes). Thus, we can understand whether and how the predicted learning processes are represented at the neural level by estimating their evolution and correlating them with neural data. Here, we present a bayesian model approach to estimate the evolution of the internal learning representations from the observations of the experiment (state estimation), and to identify the set of models' parameters (parameter estimation) and the class of behavioral model (model selection) that are most likely to have generated a given sequence of actions and outcomes. More precisely, we use Sequential Monte Carlo methods for state estimation and the maximum likelihood principle (MLP) for model selection and parameter estimation. We show that the method recovers simulated trajectories of learning sessions on a single-trial basis and provides predictions about the activity of different categories of neurons that should participate in the learning process. By correlating the estimated evolutions of the learning variables, we will be able to test the validity of different models of instrumental learning and possibly identify the neural bases of learning.

Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement

Ventral tegmental area (VTA) neuron firing precedes behaviors elicited by reward-predictive sensory cues and scales with the magnitude and unpredictability of received rewards. These patterns are consistent with roles in the performance of learned appetitive behaviors and in positive reinforcement, respectively. The VTA includes subpopulations of neurons with different afferent connections, neurotransmitter content, and projection targets. Because the VTA and substantia nigra pars compacta are the sole sources of striatal and limbic forebrain dopamine, measurements of dopamine release and manipulations of dopamine function have provided critical evidence supporting a VTA contribution to these functions. However, the VTA also sends GABAergic and glutamatergic projections to the nucleus accumbens and prefrontal cortex. Furthermore, VTA-mediated but dopamine-independent positive reinforcement has been demonstrated. Consequently, identifying the neurotransmitter content and projection target of VTA neurons recorded in vivo will be critical for determining their contribution to learned appetitive behaviors.

Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala

Animals and humans learn to approach and acquire pleasant stimuli and to avoid or defend against aversive ones. However, both pleasant and aversive stimuli can elicit arousal and attention, and their salience or intensity increases when they occur by surprise. Thus, adaptive behavior may require that neural circuits compute both stimulus valence--or value--and intensity. To explore how these computations may be implemented, we examined neural responses in the primate amygdala to unexpected reinforcement during learning. Many amygdala neurons responded differently to reinforcement depending upon whether or not it was expected. In some neurons, this modulation occurred only for rewards or aversive stimuli, but not both. In other neurons, expectation similarly modulated responses to both rewards and punishments. These different neuronal populations may subserve two sorts of processes mediated by the amygdala: those activated by surprising reinforcements of both valences-such as enhanced arousal and attention-and those that are valence-specific, such as fear or reward-seeking behavior.

Functional specialization of the primate frontal cortex during decision making

Economic theories of decision making are based on the principle of utility maximization, and reinforcement-learning theory provides computational algorithms that can be used to estimate the overall reward expected from alternative choices. These formal models not only account for a large range of behavioral observations in human and animal decision makers, but also provide useful tools for investigating the neural basis of decision making. Nevertheless, in reality, decision makers must combine different types of information about the costs and benefits associated with each available option, such as the quality and quantity of expected reward and required work. In this article, we put forward the hypothesis that different subdivisions of the primate frontal cortex may be specialized to focus on different aspects of dynamic decision-making processes. In this hypothesis, the lateral prefrontal cortex is primarily involved in maintaining the state representation necessary to identify optimal actions in a given environment. In contrast, the orbitofrontal cortex and the anterior cingulate cortex might be primarily involved in encoding and updating the utilities associated with different sensory stimuli and alternative actions, respectively. These cortical areas are also likely to contribute to decision making in a social context.

Understanding neural coding through the model-based analysis of decision making

The study of decision making poses new methodological challenges for systems neuroscience. Whereas our traditional approach linked neural activity to external variables that the experimenter directly observed and manipulated, many of the key elements that contribute to decisions are internal to the decider. Variables such as subjective value or subjective probability may be influenced by experimental conditions and manipulations but can neither be directly measured nor precisely controlled. Pioneering work on the neural basis of decision circumvented this difficulty by studying behavior in static conditions, in which knowledge of the average state of these quantities was sufficient. More recently, a new wave of studies has confronted the conundrum of internal decision variables more directly by leveraging quantitative behavioral models. When these behavioral models are successful in predicting a subject's choice, the model's internal variables may serve as proxies for the unobservable decision variables that actually drive behavior. This new methodology has allowed researchers to localize neural subsystems that encode hidden decision variables related to free choice and to study these variables under dynamic conditions.

The relationship between the subjective and objective aspects of visual filling-in

We explored the relationship between filling-in processes and the known increase in detection sensitivity observed for targets presented between collinear flankers. Filling-in was probed using a Yes/No detection task by measuring the false-positive reports (false-alarm, FA) and hit rate (Hit) for a low-contrast Gabor target with different target-flankers distances. Observers increased the number of reports on the presence of a target (FA and Hit) when the flankers' distance was within the known range of facilitatory lateral interactions. This bias in reporting was reduced with blocked stimulation, when the target-flanker distance was kept fixed across trials. When different distances were mixed by trials the bias followed the pattern of lateral interactions across distance. The effect was maximal when flankers and targets were aligned. These false perceptions are most likely the result of a filling-in process by lateral excitation that produces illusory contours.

Humans trade off viewing time and movement duration to improve visuomotor accuracy in a fast reaching task

Previous research has shown that the brain uses statistical knowledge of both sensory and motor accuracy to optimize behavioral performance. Here, we present the results of a novel experiment in which participants could control both of these quantities at once. Specifically, maximum performance demanded the simultaneous choices of viewing and movement durations, which directly impacted visual and motor accuracy. Participants reached to a target indicated imprecisely by a two-dimensional distribution of dots within a 1200 ms time limit. By choosing when to reach, participants selected the quality of visual information regarding target location as well as the remaining time available to execute the reach. New dots, and consequently more visual information, appeared until the reach was initiated; after reach initiation, no new dots appeared. However, speed accuracy trade-offs in motor control make early reaches (much remaining time) precise and late reaches (little remaining time) imprecise. Based on each participant's visual- and motor-only target-hitting performances, we computed an "ideal reacher" that selects reach initiation times that minimize predicted reach endpoint deviations from the true target location. The participant's timing choices were qualitatively consistent with ideal predictions: choices varied with stimulus changes (but less than the predicted magnitude) and resulted in near-optimal performance despite the absence of direct feedback defining ideal performance. Our results suggest visual estimates, and their respective accuracies are passed to motor planning systems, which in turn predict the precision of potential reaches and control viewing and movement timing to favorably trade off visual and motor accuracy.

The Neural Basis of Decision Making

The study of decision making spans such varied fields as neuroscience, psychology, economics, statistics, political science, and computer science. Despite this diversity of applications, most decisions share common elements including deliberation and commitment. Here we evaluate recent progress in understanding how these basic elements of decision formation are implemented in the brain. We focus on simple decisions that can be studied in the laboratory but emphasize general principles likely to extend to other settings.

Expected value, reward outcome, and temporal difference error representations in a probabilistic decision task

In probabilistic decision tasks, an expected value (EV) of a choice is calculated, and after the choice has been made, this can be updated based on a temporal difference (TD) prediction error between the EV and the reward magnitude (RM) obtained. The EV is measured as the probability of obtaining a reward x RM. To understand the contribution of different brain areas to these decision-making processes, functional magnetic resonance imaging activations related to EV versus RM (or outcome) were measured in a probabilistic decision task. Activations in the medial orbitofrontal cortex were correlated with both RM and with EV and confirmed in a conjunction analysis to extend toward the pregenual cingulate cortex. From these representations, TD reward prediction errors could be produced. Activations in areas that receive from the orbitofrontal cortex including the ventral striatum, midbrain, and inferior frontal gyrus were correlated with the TD error. Activations in the anterior insula were correlated negatively with EV, occurring when low reward outcomes were expected, and also with the uncertainty of the reward, implicating this region in basic and crucial decision-making parameters, low expected outcomes, and uncertainty.

Dopamine-Mushroom Body Circuit Regulates Saliency-Based Decision-Making in Drosophila

Drosophila melanogaster can make appropriate choices among alternative flight options on the basis of the relative salience of competing visual cues. We show that this choice behavior consists of early and late phases; the former requires activation of the dopaminergic system and mushroom bodies, whereas the latter is independent of these activities. Immunohistological analysis showed that mushroom bodies are densely innervated by dopaminergic axons. Thus, the circuit from the dopamine system to mushroom bodies is crucial for choice behavior in Drosophila.

Developmental differences in posterior mesofrontal cortex recruitment by risky rewards

Might increased risk taking in adolescence result in part from underdeveloped conflict-monitoring circuitry in the posterior mesofrontal cortex (PMC)? Adults and adolescents underwent functional magnetic resonance imaging during a monetary game of "chicken." As subjects watched ostensible winnings increase over time, they decided when to press a button to bank their winnings, knowing that if they did not stop pursuing money reward before a secret varying time limit, they would "bust" and either lose the money accrued on the current trial (low-penalty trials) or forfeit trial winnings plus a portion of previous winnings (high-penalty trials). Reward accrual at risk of low penalty (contrasted with guaranteed reward) activated the PMC in adults but not in adolescents. Across all subjects, this activation (1) correlated positively with age but negatively with risk exposure and (2) was greater when subjects busted on the previous low-penalty trial. Reward accrual at risk of high penalty was terminated sooner and recruited the PMC in both adults and adolescents when contrasted with guaranteed reward. Predecision PMC activation in the high-penalty trials was significantly reduced in trials when subjects busted. These data suggest that (1) under threat of an explicit severe penalty, recruitment of the PMC is similar in adolescents and adults and correlates with error avoidance, and (2) when potential penalties for a rewarding behavior are mild enough to encourage some risk taking, predecision PMC activation by a reward/risk conflict is sensitive to previous error outcomes, predictive of risk-aversive behavior in that trial, and underactive in adolescents.

Probabilistic reasoning by neurons

Our brains allow us to reason about alternatives and to make choices that are likely to pay off. Often there is no one correct answer, but instead one that is favoured simply because it is more likely to lead to reward. A variety of probabilistic classification tasks probe the covert strategies that humans use to decide among alternatives based on evidence that bears only probabilistically on outcome. Here we show that rhesus monkeys can also achieve such reasoning. We have trained two monkeys to choose between a pair of coloured targets after viewing four shapes, shown sequentially, that governed the probability that one of the targets would furnish reward. Monkeys learned to combine probabilistic information from the shape combinations. Moreover, neurons in the parietal cortex reveal the addition and subtraction of probabilistic quantities that underlie decision-making on this task.

The neurobiology of punishment

Animals, in particular humans, frequently punish other individuals who behave negatively or uncooperatively towards them. In animals, this usually serves to protect the personal interests of the individual concerned, and its kin. However, humans also punish altruistically, in which the act of punishing is personally costly. The propensity to do so has been proposed to reflect the cultural acquisition of norms of behaviour, which incorporates the desire to uphold equity and fairness, and promotes cooperation. Here, we review the proximate neurobiological basis of punishment, considering the motivational processes that underlie punishing actions.

Spinal interneurons that are selectively activated during fictive flexion reflex

Behavioral choices in invertebrates are mediated by a combination of shared and specialized circuitry, including neurons that are inhibited during competing behaviors. Less is known, however, about the neural mechanisms of behavioral choice in vertebrates. The spinal cord can appropriately select among several types of limb movements, including limb withdrawal (flexion reflex), scratching, and locomotion, and thus is conducive to examination of vertebrate mechanisms of behavioral choice. Flexion reflex can interrupt and reset the rhythm of scratching and locomotion, suggesting that a combination of shared and specialized circuitry contributes to these behaviors, but little is known about the interneurons involved. Here, I used in vivo intracellular recording and dye injection to identify a group of spinal interneurons that are strongly activated during fictive flexion reflex but inhibited during fictive scratching and fictive swimming. These flexion-selective interneurons are typically rhythmically hyperpolarized during fictive scratching and fictive swimming. This hyperpolarization can be maximal during the ipsilateral hip flexor bursts of rhythmic limb motor patterns, although these cells are strongly activated during the ipsilateral hip flexor bursts of fictive flexion reflex. Thus, these interneurons are relatively specialized for fictive limb withdrawal, rather than contributing to the hip flexor phase of multiple types of limb movements. These flexion-selective cells are physiologically and morphologically distinguishable from a recently described group of spinal interneurons (transverse interneurons) that are strongly activated during both fictive flexion reflex and fictive scratching. Thus, spinal interneurons with distinct behavioral roles may to some extent be morphologically distinguishable.

Fundamental components of attention

A mechanistic understanding of attention is necessary for the elucidation of the neurobiological basis of conscious experience. This chapter presents a framework for thinking about attention that facilitates the analysis of this cognitive process in terms of underlying neural mechanisms. Four processes are fundamental to attention: working memory, top-down sensitivity control, competitive selection, and automatic bottom-up filtering for salient stimuli. Each process makes a distinct and essential contribution to attention. Voluntary control of attention involves the first three processes (working memory, top-down sensitivity control, and competitive selection) operating in a recurrent loop. Recent results from neurobiological research on attention are discussed within this framework.

Model-based fMRI and its application to reward learning and decision making

In model-based functional magnetic resonance imaging (fMRI), signals derived from a computational model for a specific cognitive process are correlated against fMRI data from subjects performing a relevant task to determine brain regions showing a response profile consistent with that model. A key advantage of this technique over more conventional neuroimaging approaches is that model-based fMRI can provide insights into how a particular cognitive process is implemented in a specific brain area as opposed to merely identifying where a particular process is located. This review will briefly summarize the approach of model-based fMRI, with reference to the field of reward learning and decision making, where computational models have been used to probe the neural mechanisms underlying learning of reward associations, modifying action choice to obtain reward, as well as in encoding expected value signals that reflect the abstract structure of a decision problem. Finally, some of the limitations of this approach will be discussed.

Advances in understanding ventromedial prefrontal function

Studies of the brain basis of decision-making and economic behavior are providing a new perspective on the organization and functions of human prefrontal cortex. This line of inquiry has focused particularly on the ventral and medial portions of prefrontal cortex, arguably the most enigmatic regions of the “enigmatic frontal lobes.” This review highlights recent advances in the cognitive neuroscience of decision making and neuroeconomics and discusses how these findings can inform clinical thinking about frontal lobe dysfunction.