Choosing the greater of two goods: neural currencies for valuation and decision making

Looking time predicts choice but not aesthetic value

Although visual fixations are commonly used to index stimulus-driven or internally-determined preference, recent evidence suggests that visual fixations can also be a source of decisional bias that moves selection toward the fixated object. These contrasting results raise the question of whether visual fixations always index comparative processes during choice-based tasks, or whether they might better reflect internal preferences when the decision does not carry any economic or corporeal consequences. In two experiments, participants chose which of two objects were more aesthetically pleasing (Exp.1) or appeared more organic (Exp.2), and provided independent aesthetic ratings of the stimuli. Our results demonstrated that fixation parameters were a better index of choice in both decisional domains than of aesthetic preference. The data support models in which visual fixations are specifically related to the evolution of decision processes even when the decision has no tangible consequences.

Categorization = decision making + generalization

We rarely, if ever, repeatedly encounter exactly the same situation. This makes generalization crucial for real world decision making. We argue that categorization, the study of generalizable representations, is a type of decision making, and that categorization learning research would benefit from approaches developed to study the neuroscience of decision making. Similarly, methods developed to examine generalization and learning within the field of categorization may enhance decision making research. We first discuss perceptual information processing and integration, with an emphasis on accumulator models. We then examine learning the value of different decision making choices via experience, emphasizing reinforcement learning modeling approaches. Next we discuss how value is combined with other factors in decision making, emphasizing the effects of uncertainty. Finally, we describe how a final decision is selected via thresholding processes implemented by the basal ganglia and related regions. We also consider how memory related functions in the hippocampus may be integrated with decision making mechanisms and contribute to categorization.

Neural changes with tactile learning reflect decision-level reweighting of perceptual readout

Despite considerable work, the neural basis of perceptual learning remains uncertain. For visual learning, although some studies suggested that changes in early sensory representations are responsible, other studies point to decision-level reweighting of perceptual readout. These competing possibilities have not been examined in other sensory systems, investigating which could help resolve the issue. Here we report a study of human tactile microspatial learning in which participants achieved >six-fold decline in acuity threshold after multiple training sessions. Functional magnetic resonance imaging was performed during performance of the tactile microspatial task and a control, tactile temporal task. Effective connectivity between relevant brain regions was estimated using multivariate, autoregressive models of hidden neuronal variables obtained by deconvolution of the hemodynamic response. Training-specific increases in task-selective activation assessed using the task × session interaction and associated changes in effective connectivity primarily involved subcortical and anterior neocortical regions implicated in motor and/or decision processes, rather than somatosensory cortical regions. A control group of participants tested twice, without intervening training, exhibited neither threshold improvement nor increases in task-selective activation. Our observations argue that neuroplasticity mediating perceptual learning occurs at the stage of perceptual readout by decision networks. This is consonant with the growing shift away from strictly modular conceptualization of the brain toward the idea that complex network interactions underlie even simple tasks. The convergence of our findings on tactile learning with recent studies of visual learning reconciles earlier discrepancies in the literature on perceptual learning.

Optogenetic dissection of neural circuits underlying emotional valence and motivated behaviors

The neural circuits underlying emotional valence and motivated behaviors are several synapses away from both defined sensory inputs and quantifiable motor outputs. Electrophysiology has provided us with a suitable means for observing neural activity during behavior, but methods for controlling activity for the purpose of studying motivated behaviors have been inadequate: electrical stimulation lacks cellular specificity and pharmacological manipulation lacks temporal resolution. The recent emergence of optogenetic tools provides a new means for establishing causal relationships between neural activity and behavior. Optogenetics, the use of genetically-encodable light-activated proteins, permits the modulation of specific neural circuit elements with millisecond precision. The ability to control individual cell types, and even projections between distal regions, allows us to investigate functional connectivity in a causal manner. The greatest consequence of controlling neural activity with finer precision has been the characterization of individual neural circuits within anatomical brain regions as defined functional units. Within the mesolimbic dopamine system, optogenetics has helped separate subsets of dopamine neurons with distinct functions for reward, aversion and salience processing, elucidated GABA neuronal effects on behavior, and characterized connectivity with forebrain and cortical structures. Within the striatum, optogenetics has confirmed the opposing relationship between direct and indirect pathway medium spiny neurons (MSNs), in addition to characterizing the inhibition of MSNs by cholinergic interneurons. Within the hypothalamus, optogenetics has helped overcome the heterogeneity in neuronal cell-type and revealed distinct circuits mediating aggression and feeding. Within the amygdala, optogenetics has allowed the study of intra-amygdala microcircuitry as well as interconnections with distal regions involved in fear and anxiety. In this review, we will present the body of optogenetic studies that has significantly enhanced our understanding of emotional valence and motivated behaviors. This article is part of a Special Issue entitled Optogenetics (7th BRES).

Social information signaling by neurons in primate striatum

Social decisions depend on reliable information about others. Consequently, social primates are motivated to acquire information about the identity, social status, and reproductive quality of others. Neurophysiological and neuroimaging studies implicate the striatum in the motivational control of behavior. Neuroimaging studies specifically implicate the ventromedial striatum in signaling motivational aspects of social interaction. Despite this evidence, precisely how striatal neurons encode social information remains unknown. Therefore, we probed the activity of single striatal neurons in monkeys choosing between visual social information at the potential expense of fluid reward. We show for the first time that a population of neurons located primarily in medial striatum selectively signals social information. Surprisingly, representation of social information was unrelated to simultaneously expressed social preferences. A largely nonoverlapping population of neurons that was not restricted to the medial striatum signaled information about fluid reward. Our findings demonstrate that information about social context and nutritive reward are maintained largely independently in striatum, even when both influence decisions to execute a single action.

Working memory and decision processes in visual area v4

Recognizing and responding to a remembered stimulus requires the coordination of perception, working memory, and decision-making. To investigate the role of visual cortex in these processes, we recorded responses of single V4 neurons during performance of a delayed match-to-sample task that incorporates rapid serial visual presentation of natural images. We found that neuronal activity during the delay period after the cue but before the images depends on the identity of the remembered image and that this change persists while distractors appear. This persistent response modulation has been identified as a diagnostic criterion for putative working memory signals; our data thus suggest that working memory may involve reactivation of sensory neurons. When the remembered image reappears in the neuron’s receptive field, visually evoked responses are enhanced; this match enhancement is a diagnostic criterion for decision. One model that predicts these data is the matched filter hypothesis, which holds that during search V4 neurons change their tuning so as to match the remembered cue, and thus become detectors for that image. More generally, these results suggest that V4 neurons participate in the perceptual, working memory, and decision processes that are needed to perform memory-guided decision-making.

The primate amygdala combines information about space and value

A stimulus predicting reinforcement can trigger emotional responses, such as arousal, as well as cognitive ones, such as increasing attention towards that stimulus. Neuroscientists have long appreciated that the amygdala mediates spatially non-specific emotional responses, but it remains unclear whether the amygdala links motivational and spatial representations. To test whether amygdala neurons encode spatial and motivational information, we presented reward-predictive cues in different spatial configurations while assessing whether these cues influenced spatial attention. Cue configuration and predicted reward magnitude modulated amygdala neural activity in a coordinated fashion. Moreover, fluctuations in activity were correlated with trial-to-trial variability in spatial attention. Thus the amygdala integrates spatial and motivational information, which may influence the spatial allocation of cognitive resources. These results suggest that amygdala dysfunction may contribute to deficits in cognitive processes normally coordinated with emotional responses, such as directing attention towards the location of emotionally-relevant stimuli.

Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcements driving both action acquisition and reward maximization: a simulated robotic study

An important issue of recent neuroscientific research is to understand the functional role of the phasic release of dopamine in the striatum, and in particular its relation to reinforcement learning. The literature is split between two alternative hypotheses: one considers phasic dopamine as a reward prediction error similar to the computational TD-error, whose function is to guide an animal to maximize future rewards; the other holds that phasic dopamine is a sensory prediction error signal that lets the animal discover and acquire novel actions. In this paper we propose an original hypothesis that integrates these two contrasting positions: according to our view phasic dopamine represents a TD-like reinforcement prediction error learning signal determined by both unexpected changes in the environment (temporary, intrinsic reinforcements) and biological rewards (permanent, extrinsic reinforcements). Accordingly, dopamine plays the functional role of driving both the discovery and acquisition of novel actions and the maximization of future rewards. To validate our hypothesis we perform a series of experiments with a simulated robotic system that has to learn different skills in order to get rewards. We compare different versions of the system in which we vary the composition of the learning signal. The results show that only the system reinforced by both extrinsic and intrinsic reinforcements is able to reach high performance in sufficiently complex conditions.

Neural coding of computational factors affecting decision making

We constantly need to make decisions that can result in rewards of different amounts with different probabilities and at different timing. To characterize the neural coding of such computational factors affecting value-based decision making, we have investigated how reward information processing is influenced by parameters such as reward magnitude, probability, delay, effort, and uncertainty using either fMRI in healthy humans or intracranial recordings in patients with epilepsy. We decomposed brain signals modulated by these computational factors, showing that prediction error (PE), salient PE, and uncertainty signals are computed in partially overlapping brain circuits and that both transient and sustained uncertainty signals coexist in the brain. When investigating the neural representation of primary and secondary rewards, we found both a common brain network, including the ventromedial prefrontal cortex and ventral striatum, and a functional organization of the orbitofrontal cortex according to reward type. Moreover, separate valuation systems were engaged for delay and effort costs when deciding between options. Finally, genetic variations in dopamine-related genes influenced the response of the reward system and may contribute to individual differences in reward-seeking behavior and in predisposition to neuropsychiatric disorders.

Phasic transition from goal-directed to habitual control over drug-seeking produced by conflicting reinforcer expectancy

The transition from goal-directed to habitual control over drug-seeking has been experimentally demonstrated in animals, but there have been no comparable reports in humans. Following a recent animal design, the current study employed an outcome-devaluation procedure to test whether goal-directed control over tobacco seeking would be abolished by alcohol expectancy. Eighty smokers first learned that two responses earned tobacco or chocolate points, respectively, before tobacco was devalued by health warnings and smoking satiety. Participants were then presented with either a glass of beer/wine or water with instructions that this item could be consumed after the task (alternative reward). Then choice between the tobacco and chocolate response was measured in extinction to assess goal-directed control of tobacco seeking, in a nominal Pavlovian to instrumental transfer (PIT) test to assess stimulus control of tobacco seeking, and in a reacquisition test to assess the impact of direct feedback from the outcomes. The results showed that alcohol expectancy selectively abolished goal-directed control of tobacco seeking but not stimulus control or the impact of feedback from outcomes. These data suggest that 'endogenous' retrieval of low drug value governing goal-directed regulation of drug seeking is disrupted by conflicting appraisal of an alternative reinforcer, promoting habitual control, which may play a role in relapse.

Confidence in value-based choice

Decisions are never perfect, with confidence in one's choices fluctuating over time. How subjective confidence and valuation of choice options interact at the level of brain and behavior is unknown. Using a dynamic model of the decision process, we show that confidence reflects the evolution of a decision variable over time, explaining the observed relation between confidence, value, accuracy and reaction time. As predicted by our dynamic model, we show that a functional magnetic resonance imaging signal in human ventromedial prefrontal cortex (vmPFC) reflects both value comparison and confidence in the value comparison process. Crucially, individuals varied in how they related confidence to accuracy, allowing us to show that this introspective ability is predicted by a measure of functional connectivity between vmPFC and rostrolateral prefrontal cortex. Our findings provide a mechanistic link between noise in value comparison and metacognitive awareness of choice, enabling us both to want and to express knowledge of what we want.

Rewards teach visual selective attention

Visual selective attention is the brain function that modulates ongoing processing of retinal input in order for selected representations to gain privileged access to perceptual awareness and guide behavior. Enhanced analysis of currently relevant or otherwise salient information is often accompanied by suppressed processing of the less relevant or salient input. Recent findings indicate that rewards exert a powerful influence on the deployment of visual selective attention. Such influence takes different forms depending on the specific protocol adopted in the given study. In some cases, the prospect of earning a larger reward in relation to a specific stimulus or location biases attention accordingly in order to maximize overall gain. This is mediated by an effect of reward acting as a type of incentive motivation for the strategic control of attention. In contrast, reward delivery can directly alter the processing of specific stimuli by increasing their attentional priority, and this can be measured even when rewards are no longer involved, reflecting a form of reward-mediated attentional learning. As a further development, recent work demonstrates that rewards can affect attentional learning in dissociable ways depending on whether rewards are perceived as feedback on performance or instead are registered as random-like events occurring during task performance. Specifically, it appears that visual selective attention is shaped by two distinct reward-related learning mechanisms: one requiring active monitoring of performance and outcome, and a second one detecting the sheer association between objects in the environment (whether attended or ignored) and the more-or-less rewarding events that accompany them. Overall this emerging literature demonstrates unequivocally that rewards "teach" visual selective attention so that processing resources will be allocated to objects, features and locations which are likely to optimize the organism's interaction with the surrounding environment and maximize positive outcome.

Neuroethology of decision-making

A neuroethological approach to decision-making considers the effect of evolutionary pressures on neural circuits mediating choice. In this view, decision systems are expected to enhance fitness with respect to the local environment, and particularly efficient solutions to specific problems should be conserved, expanded, and repurposed to solve other problems. Here, we discuss basic prerequisites for a variety of decision systems from this viewpoint. We focus on two of the best-studied and most widely represented decision problems. First, we examine patch leaving, a prototype of environmentally based switching between action patterns. Second, we consider social information seeking, a process resembling foraging with search costs. We argue that while the specific neural solutions to these problems sometimes differ across species, both the problems themselves and the algorithms instantiated by biological hardware are repeated widely throughout nature. The behavioral and mathematical study of ubiquitous decision processes like patch leaving and social information seeking thus provides a powerful new approach to uncovering the fundamental design structure of nervous systems.

Coding of the long-term value of multiple future rewards in the primate striatum

Decisions maximizing benefits involve a tradeoff between the quantity of a reward and the cost of elapsed time until an animal receives it. The estimation of long-term reward values is critical to attain the most desirable outcomes over a certain period of time. Reinforcement learning theories have established algorithms to estimate the long-term reward values of multiple future rewards in which the values of future rewards are discounted as a function of how many steps of choices are necessary to achieve them. Here, we report that presumed striatal projection neurons represent the long-term values of multiple future rewards estimated by a standard reinforcement learning model while monkeys are engaged in a series of trial-and-error choices and adaptive decisions for multiple rewards. We found that the magnitude of activity of a subset of neurons was positively correlated with the long-term reward values, and that of another subset of neurons was negatively correlated throughout the entire decision-making process in individual trials: from the start of the task trial, estimation of the values and their comparison among alternatives, choice execution, and evaluation of the received rewards. An idiosyncratic finding was that neurons showing negative correlations represented reward values in the near future (high discounting), while neurons showing positive correlations represented reward values not only in the near future, but also in the far future (low discounting). These findings provide a new insight that long-term value signals are embedded in two subsets of striatal neurons as high and low discounting of multiple future rewards.

Building a bridge into the future: dynamic connectionist modeling as an integrative tool for research on intertemporal choice

Temporal discounting denotes the fact that individuals prefer smaller rewards delivered sooner over larger rewards delivered later, often to a higher extent than suggested by normative economical theories. In this article, we identify three lines of research studying this phenomenon which aim (i) to describe temporal discounting mathematically, (ii) to explain observed choice behavior psychologically, and (iii) to predict the influence of specific factors on intertemporal decisions. We then opt for an approach integrating postulated mechanisms and empirical findings from these three lines of research. Our approach focuses on the dynamical properties of decision processes and is based on computational modeling. We present a dynamic connectionist model of intertemporal choice focusing on the role of self-control and time framing as two central factors determining choice behavior. Results of our simulations indicate that the two influences interact with each other, and we present experimental data supporting this prediction. We conclude that computational modeling of the decision process dynamics can advance the integration of different strands of research in intertemporal choice.

Sensitivity to fairness and intentions of others in the ultimatum game in patients with ventromedial prefontal lesions

This study aimed to examine the relationship between perspective-taking and impaired decision-making in patients with ventromedial prefrontal (VM) lesions, using the Ultimatum Game (UG). In the UG, two players split a sum of money and one player proposes a division while the other can accept or reject this. Eight patients with VM damage and 18 healthy controls participated as responders in a modified version of the UG, in which identical offers can generate different rejection rates depending on the other offers available to the proposer. Participants had to either accept or reject offers of 2:8 NIS (2NIS for them and 8 NIS for the proposer), which were paired with one of four different possible offers (5:5, 4:6, 2:8, 8:2). Results indicate that the controls more often rejected offers of 2:8 when the alternative was 4:6 (a greedy alternative) than when the alternative was 5:5 (fair alternative), whereas the VM patients showed the opposite pattern of decision-making. Additionally, the overall rejection rates were higher in patients as compared to controls. Furthermore, scores on a perspective-taking scale were negatively correlated with rejection rates in the patient group, suggesting that perspective-taking deficits may account for impaired decision-making in VM patients.

Attention, learning, and the value of information

Despite many studies on selective attention, fundamental questions remain about its nature and neural mechanisms. Here I draw from the animal and machine learning fields that describe attention as a mechanism for active learning and uncertainty reduction and explore the implications of this view for understanding visual attention and eye movement control. I propose that a closer integration of these different views has the potential greatly to expand our understanding of oculomotor control and our ability to use this system as a window into high level but poorly understood cognitive functions, including the capacity for curiosity and exploration and for inferring internal models of the external world.

In monkeys making value-based decisions, LIP neurons encode cue salience and not action value

In monkeys deciding between alternative saccadic eye movements, lateral intraparietal (LIP) neurons representing each saccade fire at a rate proportional to the value of the reward expected upon its completion. This observation has been interpreted as indicating that LIP neurons encode saccadic value and that they mediate value-based decisions between saccades. Here, we show that LIP neurons representing a given saccade fire strongly not only if it will yield a large reward but also if it will incur a large penalty. This finding indicates that LIP neurons are sensitive to the motivational salience of cues. It is compatible neither with the idea that LIP neurons represent action value nor with the idea that value-based decisions take place in LIP neurons.

When less is more: evolutionary origins of the affect heuristic

The human mind is built for approximations. When considering the value of a large aggregate of different items, for example, we typically do not summate the many individual values. Instead, we appear to form an immediate impression of the likeability of the option based on the average quality of the full collection, which is easier to evaluate and remember. While useful in many situations, this affect heuristic can lead to apparently irrational decision-making. For example, studies have shown that people are willing to pay more for a small set of high-quality goods than for the same set of high-quality goods with lower-quality items added [e.g. 1]. We explored whether this kind of choice behavior could be seen in other primates. In two experiments, one in the laboratory and one in the field, using two different sets of food items, we found that rhesus monkeys preferred a highly-valued food item alone to the identical item paired with a food of positive but lower value. This finding provides experimental evidence that, under certain conditions, macaque monkeys follow an affect heuristic that can cause them to prefer less food. Conservation of this affect heuristic could account for similar ‘irrational’ biases in humans, and may reflect a more general complexity reduction strategy in which averages, prototypes, or stereotypes represent a set or group.

Columnar processing in primate pFC: evidence for executive control microcircuits

A common denominator for many cognitive disorders of human brain is the disruption of neural activity within pFC, whose structural basis is primarily interlaminar (columnar) microcircuits or "minicolumns." The importance of this brain region for executive decision-making has been well documented; however, because of technological constraints, the minicolumnar basis is not well understood. Here, via implementation of a unique conformal multielectrode recording array, the role of interlaminar pFC minicolumns in the executive control of task-related target selection is demonstrated in nonhuman primates performing a visuomotor DMS task. The results reveal target-specific, interlaminar correlated firing during the decision phase of the trial between multielectrode recording array-isolated minicolumnar pairs of neurons located in parallel in layers 2/3 and layer 5 of pFC. The functional significance of individual pFC minicolumns (separated by 40 μm) was shown by reduced correlated firing between cell pairs within single minicolumns on error trials with inappropriate target selection. To further demonstrate dependence on performance, a task-disrupting drug (cocaine) was administered in the middle of the session, which also reduced interlaminar firing in minicolumns that fired appropriately in the early (nondrug) portion of the session. The results provide a direct demonstration of task-specific, real-time columnar processing in pFC indicating the role of this type of microcircuit in executive control of decision-making in primate brain.

Cognitive regulation during decision making shifts behavioral control between ventromedial and dorsolateral prefrontal value systems

Cognitive regulation is often used to influence behavioral outcomes. However, the computational and neurobiological mechanisms by which it affects behavior remain unknown. We studied this issue using an fMRI task in which human participants used cognitive regulation to upregulate and downregulate their cravings for foods at the time of choice. We found that activity in both ventromedial prefrontal cortex (vmPFC) and dorsolateral prefrontal cortex (dlPFC) correlated with value. We also found evidence that two distinct regulatory mechanisms were at work: value modulation, which operates by changing the values assigned to foods in vmPFC and dlPFC at the time of choice, and behavioral control modulation, which operates by changing the relative influence of the vmPFC and dlPFC value signals on the action selection process used to make choices. In particular, during downregulation, activation decreased in the value-sensitive region of dlPFC (indicating value modulation) but not in vmPFC, and the relative contribution of the two value signals to behavior shifted toward the dlPFC (indicating behavioral control modulation). The opposite pattern was observed during upregulation: activation increased in vmPFC but not dlPFC, and the relative contribution to behavior shifted toward the vmPFC. Finally, ventrolateral PFC and posterior parietal cortex were more active during both upregulation and downregulation, and were functionally connected with vmPFC and dlPFC during cognitive regulation, which suggests that they help to implement the changes to the decision-making circuitry generated by cognitive regulation.

Facilitation and restoration of cognitive function in primate prefrontal cortex by a neuroprosthesis that utilizes minicolumn-specific neural firing

OBJECTIVE

Maintenance of cognitive control is a major concern for many human disease conditions; therefore, a major goal of human neuroprosthetics is to facilitate and/or recover the cognitive function when such circumstances impair appropriate decision making.

APPROACH

Minicolumnar activity from the prefrontal cortex (PFC) was recorded from nonhuman primates trained to perform a delayed match to sample (DMS), via custom-designed conformal multielectrode arrays that provided inter-laminar recordings from neurons in the PFC layer 2/3 and layer 5. Such recordings were analyzed via a previously demonstrated nonlinear multi-input-multi-output (MIMO) neuroprosthesis in rodents, which extracted and characterized multicolumnar firing patterns during DMS performance.

MAIN RESULTS

The MIMO model verified that the conformal recorded individual PFC minicolumns responded to entrained target selections in patterns critical for successful DMS performance. This allowed the substitution of task-related layer 5 neuron firing patterns with electrical stimulation in the same recording regions during columnar transmission from layer 2/3 at the time of target selection. Such stimulation improved normal task performance, but more importantly, recovered performance when applied as a neuroprosthesis following the pharmacological disruption of decision making in the same task.

SIGNIFICANCE

These findings provide the first successful application of neuroprosthesis in the primate brain designed specifically to restore or repair the disrupted cognitive function.

Decision Making and Behavioral Choice during Predator Avoidance

One of the most important decisions animals have to make is how to respond to an attack from a potential predator. The response must be prompt and appropriate to ensure survival. Invertebrates have been important models in studying the underlying neurobiology of the escape response due to their accessible nervous systems and easily quantifiable behavioral output. Moreover, invertebrates provide opportunities for investigating these processes at a level of analysis not available in most other organisms. Recently, there has been a renewed focus in understanding how value-based calculations are made on the level of the nervous system, i.e., when decisions are made under conflicting circumstances, and the most desirable choice must be selected by weighing the costs and benefits for each behavioral choice. This article reviews samples from the current literature on anti-predator decision making in invertebrates, from single neurons to complex behaviors. Recent progress in understanding the mechanisms underlying value-based behavioral decisions is also discussed.

Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value

In changing environments, animals must adaptively select actions to achieve their goals. In tasks involving goal-directed action selection, striatal neural activity has been shown to represent the value of competing actions. Striatal representations of action value could potentially bias responses toward actions of higher value. However, no study to date has demonstrated the direct effect of distinct striatal pathways in goal-directed action selection. We found that transient optogenetic stimulation of dorsal striatal dopamine D1 and D2 receptor-expressing neurons during decision-making in mice introduced opposing biases in the distribution of choices. The effect of stimulation on choice was dependent on recent reward history and mimicked an additive change in the action value. Although stimulation before and during movement initiation produced a robust bias in choice behavior, this bias was substantially diminished when stimulation was delayed after response initiation. Together, our data suggest that striatal activity is involved in goal-directed action selection.

Making decisions through a distributed consensus

How does the brain decide between actions? Is it through comparisons of abstract representations of outcomes or through a competition in a sensorimotor map defining the actions themselves? Here, I review strengths and limitations of both of these proposals, and suggest that decisions emerge through a distributed consensus across many levels of representation.

Salience driven value integration explains decision biases and preference reversal

Human choice behavior exhibits many paradoxical and challenging patterns. Traditional explanations focus on how values are represented, but little is known about how values are integrated. Here we outline a psychophysical task for value integration that can be used as a window on high-level, multiattribute decisions. Participants choose between alternative rapidly presented streams of numerical values. By controlling the temporal distribution of the values, we demonstrate that this process underlies many puzzling choice paradoxes, such as temporal, risk, and framing biases, as well as preference reversals. These phenomena can be explained by a simple mechanism based on the integration of values, weighted by their salience. The salience of a sampled value depends on its temporal order and momentary rank in the decision context, whereas the direction of the weighting is determined by the task framing. We show that many known choice anomalies may arise from the microstructure of the value integration process.

Building Bridges between Perceptual and Economic Decision-Making: Neural and Computational Mechanisms

Investigation into the neural and computational bases of decision-making has proceeded in two parallel but distinct streams. Perceptual decision-making (PDM) is concerned with how observers detect, discriminate, and categorize noisy sensory information. Economic decision-making (EDM) explores how options are selected on the basis of their reinforcement history. Traditionally, the sub-fields of PDM and EDM have employed different paradigms, proposed different mechanistic models, explored different brain regions, disagreed about whether decisions approach optimality. Nevertheless, we argue that there is a common framework for understanding decisions made in both tasks, under which an agent has to combine sensory information (what is the stimulus) with value information (what is it worth). We review computational models of the decision process typically used in PDM, based around the idea that decisions involve a serial integration of evidence, and assess their applicability to decisions between good and gambles. Subsequently, we consider the contribution of three key brain regions – the parietal cortex, the basal ganglia, and the orbitofrontal cortex (OFC) – to perceptual and EDM, with a focus on the mechanisms by which sensory and reward information are integrated during choice. We find that although the parietal cortex is often implicated in the integration of sensory evidence, there is evidence for its role in encoding the expected value of a decision. Similarly, although much research has emphasized the role of the striatum and OFC in value-guided choices, they may play an important role in categorization of perceptual information. In conclusion, we consider how findings from the two fields might be brought together, in order to move toward a general framework for understanding decision-making in humans and other primates.

A computational framework for the study of confidence in humans and animals

Confidence judgements, self-assessments about the quality of a subject's knowledge, are considered a central example of metacognition. Prima facie, introspection and self-report appear the only way to access the subjective sense of confidence or uncertainty. Contrary to this notion, overt behavioural measures can be used to study confidence judgements by animals trained in decision-making tasks with perceptual or mnemonic uncertainty. Here, we suggest that a computational approach can clarify the issues involved in interpreting these tasks and provide a much needed springboard for advancing the scientific understanding of confidence. We first review relevant theories of probabilistic inference and decision-making. We then critically discuss behavioural tasks employed to measure confidence in animals and show how quantitative models can help to constrain the computational strategies underlying confidence-reporting behaviours. In our view, post-decision wagering tasks with continuous measures of confidence appear to offer the best available metrics of confidence. Since behavioural reports alone provide a limited window into mechanism, we argue that progress calls for measuring the neural representations and identifying the computations underlying confidence reports. We present a case study using such a computational approach to study the neural correlates of decision confidence in rats. This work shows that confidence assessments may be considered higher order, but can be generated using elementary neural computations that are available to a wide range of species. Finally, we discuss the relationship of confidence judgements to the wider behavioural uses of confidence and uncertainty.

Statistical Mechanics of Reward-Modulated Learning in Decision-Making Networks

The neural substrates of decision making have been intensively studied using experimental and computational approaches. Alternative-choice tasks accompanying reinforcement have often been employed in investigations into decision making. Choice behavior has been empirically found in many experiments to follow Herrnstein's matching law. A number of theoretical studies have been done on explaining the mechanisms responsible for matching behavior. Various learning rules have been proved in these studies to achieve matching behavior as a steady state of learning processes. The models in the studies have consisted of a few parameters. However, a large number of neurons and synapses are expected to participate in decision making in the brain. We investigated learning behavior in simple but large-scale decision-making networks. We considered the covariance learning rule, which has been demonstrated to achieve matching behavior as a steady state (Loewenstein & Seung, 2006 ). We analyzed model behavior in a thermodynamic limit where the number of plastic synapses went to infinity. By means of techniques of the statistical mechanics, we can derive deterministic differential equations in this limit for the order parameters, which allow an exact calculation of the evolution of choice behavior. As a result, we found that matching behavior cannot be a steady state of learning when the fluctuations in input from individual sensory neurons are so large that they affect the net input to value-encoding neurons. This situation naturally arises when the synaptic strength is sufficiently strong and the excitatory input and the inhibitory input to the value-encoding neurons are balanced. The deviation from matching behavior is caused by increasing variance in the input potential due to the diffusion of synaptic efficacies. This effect causes an undermatching phenomenon, which has been often observed in behavioral experiments.

Pathological gambling: a systematic review of biochemical, neuroimaging, and neuropsychological findings

Pathological gambling is an emerging psychiatric disorder that has recently gained much attention because of its increasing prevalence and devastating personal, familial, and social consequences. Although its pathophysiology is largely unknown, the shared similarities with both addiction and obsessive-compulsive spectrum disorders have suggested the possibility of common psychobiological substrates. As with many other psychiatric disorders, it is believed that pathological gambling may result from the interplay between individual vulnerability and environmental factors. The aim of this article is to offer a comprehensive review of the main neurobiological aspects of pathological gambling, with particular attention to neuropsychological and related findings. A deeper understanding of the biological correlates of pathological gambling is required in order to develop effective treatment strategies.

Perceiving numbers affects the internal random movements generator

According to the evidence of direct relationships among space, numbers, and finger representations, a random movement generation (RMG) task was employed in order to investigate whether numerical exposure can influence the finger selection of healthy humans. To this purpose a group of participants were asked to generate random finger movements during the exposure to several numerical cues. Although participants were explicitly asked to move finger as random as possible, results showed that left-hand fingers were moved more frequently than right-hand fingers when low numerical cues (from 1 to 3) were presented, and, vice versa, right-hand fingers were moved more frequently than left-hand fingers when high numerical cues (ranged from 7 to 9) were presented. The current result suggests that spontaneous actions can be affected by abstract information, providing an evidence that numerical concepts can influence low-level, non-goal-directed behaviours.

The neurobiology of decision-making and responsibility: reconciling mechanism and mindedness

This essay reviews recent developments in neurobiology which are beginning to expose the mechanisms that underlie some elements of decision-making that bear on attributions of responsibility. These “elements” have been mainly studied in simple perceptual decision tasks, which are performed similarly by humans and non-human primates. Here we consider the role of neural noise, and suggest that thinking about the role of noise can shift the focus of discussions of randomness in decision-making away from its role in enabling alternate possibilities and toward a potential grounding role for responsibility.

Deliberation in the motor system: reflex gains track evolving evidence leading to a decision

Both decision making and sensorimotor control require real-time processing of noisy information streams. Historically these processes were thought to operate sequentially: cognitive processing leads to a decision, and the outcome is passed to the motor system to be converted into action. Recently, it has been suggested that the decision process may provide a continuous flow of information to the motor system, allowing it to prepare in a graded fashion for the probable outcome. Such continuous flow is supported by electrophysiology in nonhuman primates. Here we provide direct evidence for the continuous flow of an evolving decision variable to the motor system in humans. Subjects viewed a dynamic random dot display and were asked to indicate their decision about direction by moving a handle to one of two targets. We probed the state of the motor system by perturbing the arm at random times during decision formation. Reflex gains were modulated by the strength and duration of motion, reflecting the accumulated evidence in support of the evolving decision. The magnitude and variance of these gains tracked a decision variable that explained the subject's decision accuracy. The findings support a continuous process linking the evolving computations associated with decision making and sensorimotor control.

Paradoxical evidence integration in rapid decision processes

Decisions about noisy stimuli require evidence integration over time. Traditionally, evidence integration and decision making are described as a one-stage process: a decision is made when evidence for the presence of a stimulus crosses a threshold. Here, we show that one-stage models cannot explain psychophysical experiments on feature fusion, where two visual stimuli are presented in rapid succession. Paradoxically, the second stimulus biases decisions more strongly than the first one, contrary to predictions of one-stage models and intuition. We present a two-stage model where sensory information is integrated and buffered before it is fed into a drift diffusion process. The model is tested in a series of psychophysical experiments and explains both accuracy and reaction time distributions.

In models of decision making, evidence is accumulated until it crosses a threshold. The amount of evidence is directly related to the strength of the sensory input for the decision alternatives. Such one-stage models predict that if two stimulus alternatives are presented in succession, the stimulus alternative presented first dominates the decision, as the accumulated evidence will reach the threshold for this alternative first. Here, we show that for short stimulus durations decision making is not dominated by the first, but by the second stimulus. This result cannot be explained by classical one-stage decision models. We present a two-stage model where sensory input is first integrated before its outcome is fed into a classical decision process.

Representations of appetitive and aversive information in the primate orbitofrontal cortex

Individuals weigh information about both rewarding and aversive stimuli to make adaptive decisions. Most studies of the orbitofrontal cortex (OFC), an area where appetitive and aversive neural subsystems might interact, have focused only on reward. Using a classical conditioning task where novel stimuli are paired with a reward or an aversive air puff, we discovered that two groups of orbitofrontal neurons respond preferentially to conditioned stimuli associated with rewarding and aversive outcomes; however, information about appetitive and aversive stimuli converges on individual neurons from both populations. Therefore, neurons in the OFC might participate in appetitive and aversive networks that track the motivational significance of stimuli even when they vary in valence and sensory modality. Further, we show that these networks, which also extend to the amygdala, exhibit different rates of change during reversal learning. Thus, although both networks represent appetitive and aversive associations, their distinct temporal dynamics might indicate different roles in learning processes.

D1/D5 dopamine receptors modulate spatial memory formation

We investigated the effect of the intra-CA1 administration of the D1/D5 receptor antagonist SCH23390 and the D1/D5 receptor agonist SKF38393 on spatial memory in the water maze. When given immediately, but not 3h after training, SCH23390 hindered long-term spatial memory formation without affecting non-spatial memory or the normal functionality of the hippocampus. On the contrary, post-training infusion of SKF38393 enhanced retention and facilitated the spontaneous recovery of the original spatial preference after reversal learning. Our findings demonstrate that hippocampal D1/D5 receptors play an essential role in spatial memory processing.

Reward and attentional control in visual search

It has long been known that the control of attention in visual search depends both on voluntary, top-down deployment according to context-specific goals, and on involuntary, stimulus-driven capture based on the physical conspicuity of perceptual objects. Recent evidence suggests that pairing target stimuli with reward can modulate the voluntary deployment of attention, but there is little evidence that reward modulates the involuntary deployment of attention to task-irrelevant distractors. We report several experiments that investigate the role of reward learning on attentional control. Each experiment involved a training phase and a test phase. In the training phase, different colors were associated with different amounts of monetary reward. In the test phase, color was not task-relevant and participants searched for a shape singleton; in most experiments no reward was delivered in the test phase. We first show that attentional capture by physically salient distractors is magnified by a previous association with reward. In subsequent experiments we demonstrate that physically inconspicuous stimuli previously associated with reward capture attention persistently during extinction--even several days after training. Furthermore, vulnerability to attentional capture by high-value stimuli is negatively correlated across individuals with working memory capacity and positively correlated with trait impulsivity. An analysis of intertrial effects reveals that value-driven attentional capture is spatially specific. Finally, when reward is delivered at test contingent on the task-relevant shape feature, recent reward history modulates value-driven attentional capture by the irrelevant color feature. The influence of learned value on attention may provide a useful model of clinical syndromes characterized by similar failures of cognitive control, including addiction, attention-deficit/hyperactivity disorder, and obesity.

Deciding between conflicting motivations: what mice make of their prefrontal cortex

We investigated the contribution of the mouse prefrontal cortex and, more specifically, the prelimbic area, to a learning task that highlights a choice between two conflicting motivations, the one for food seeking and the one for novelty exploration. We used a learning paradigm in a cross maze task that targets first motivation conflict and second flexible rule adaptation following environmental changes. We show that mice with prelimbic lesions, which showed normal spatial learning, exhibit impairment in switching from one type of reward - food retrieval - to another type - novelty exploration - and sustained difficulties in adapting their behaviour when the rule is changing repeatedly. Mice, like other mammals, possess a prefrontal cortex that participates in the control of the flexible switch between concurrent natural motivations and in the rapid and flexible adaptation to external changes. These results open a way to study in mice models motivation conflict and cognitive adaptation, brain functions known to be compromised in several psychiatric conditions in humans in which the prefrontal cortex functioning is altered.

Leptin regulates the reward value of nutrient

We developed an assay for quantifying the reward value of nutrient and used it to analyze the effects of metabolic state and leptin. In this assay, mice chose between two sippers, one of which dispensed water and was coupled to optogenetic activation of dopaminergic (DA) neurons and the other of which dispensed natural or artificial sweeteners. This assay measured the reward value of sweeteners relative to lick-induced optogenetic activation of DA neurons. Mice preferred optogenetic stimulation of DA neurons to sucralose, but not to sucrose. However, the mice preferred sucralose plus optogenetic stimulation versus sucrose. We found that food restriction increased the value of sucrose relative to sucralose plus optogenetic stimulation, and that leptin decreased it. Our data suggest that leptin suppresses the ability of sucrose to drive taste-independent DA neuronal activation and provide new insights into the mechanism of leptin's effects on food intake.

Cortico-basal ganglia mechanisms for overcoming innate, habitual and motivational behaviors

Most of the human behaviors are executed automatically under familiar circumstances. These behaviors are prepotent in that they take precedence over any other potential alternatives. Yet, humans are also capable of engaging cognitive resources to inhibit such a prepotent behavior and replace it with an alternative controlled behavior in response to an unforeseen situation. This remarkable capability to switch behaviors in a short period of time is the hallmark of executive functions. In this article, we first argue that the prepotent automaticity could emerge at least in three different domains - innate, habitual and motivational. We then review neurophysiological findings on how the brain might realize its switching functions in each domain, primarily by focusing on the monkey oculomotor system as the experimental model. Emerging evidence now suggests that multiple neuronal populations in the shared cortico-basal ganglia network contribute to overriding prepotent eye movement, be its origin innate, habitual or motivational. This consideration suggests the general versatility of the cortico-basal ganglia network as the neural mechanism whereby humans and other animals keep themselves from becoming subservient to reflex, habit and motivational impulses.

What cognitive processes drive response biases? A diffusion model analysis

We used a diffusion model to examine the effects of response-bias manipulations on response time (RT) and accuracy data collected in two experiments involving a two-choice decision making task. We asked 18 subjects to respond “low” or “high” to the number of asterisks in a 10×10 grid, based on an experimenter-determined decision cutoff. In the model, evidence is accumulated until either a “low” or “high” decision criterion is reached, and this, in turn, initiates a response. We performed two experiments with four experimental conditions. In conditions 1 and 2, the decision cutoff between low and high judgments was fixed at 50. In condition 1, we manipulated the frequency with which low- and high-stimuli were presented. In condition 2, we used payoff structures that mimicked the frequency manipulation. We found that manipulating stimulus frequency resulted in a larger effect on RT and accuracy than did manipulating payoff structure. In the model, we found that manipulating stimulus frequency produced greater changes in the starting point of the evidence accumulation process than did manipulating payoff structure. In conditions 3 and 4, we set the decision cutoff at 40, 50, or 60 (Experiment 1) and at 45 or 55 (Experiment 2). In condition 3, there was an equal number of low- and high-stimuli, whereas in condition 4 there were unequal proportions of low- and high-stimuli. The model analyses showed that starting-point changes accounted for biases produced by changes in stimulus proportions, whereas evidence biases accounted for changes in the decision cutoff.

Competition for visual selection in the oculomotor system

During behavior, the oculomotor system is tasked with selecting objects from an ever-changing visual field and guiding eye movements to these locations. The attentional priority given to visual targets during selection can be strongly influenced by external stimulus properties or internal goals based on previous experience. Although these exogenous and endogenous drivers of selection are known to operate across partially overlapping timescales, the form of their interaction over time remains poorly understood. Using a novel choice task that simultaneously manipulates stimulus- and goal-driven attention, we demonstrate that exogenous and endogenous attentional biases change linearly as a function of time after stimulus onset and have an additive influence on the visual selection process in rhesus macaques (Macaca mulatta). We present a family of computational models that quantify this interaction over time and detail the history dependence of both processes. The computational models reveal the existence of a critical 140-180 ms attentional "switching" time, when stimulus- and goal-driven processes simultaneously favor competing visual targets. These results suggest that the brain uses a linear sum of attentional biases to guide visual selection.

Choosing goals, not rules: deciding among rule-based action plans

In natural situations, movements are often directed toward locations different from that of the evoking sensory stimulus. Movement goals must then be inferred from the sensory cue based on rules. When there is uncertainty about the rule that applies for a given cue, planning a movement involves both choosing the relevant rule and computing the movement goal based on that rule. Under these conditions, it is not clear whether primates compute multiple movement goals based on all possible rules before choosing an action, or whether they first choose a rule and then only represent the movement goal associated with that rule. Supporting the former hypothesis, we show that neurons in the frontoparietal reach areas of monkeys simultaneously represent two different rule-based movement goals, which are biased by the monkeys' choice preferences. Apparently, primates choose between multiple behavioral options by weighing against each other the movement goals associated with each option.

Human consciousness and its relationship to social neuroscience: A novel hypothesis

A common modern view of consciousness is that it is an emergent property of the brain, perhaps caused by neuronal complexity, and perhaps with no adaptive value. Exactly what emerges, how it emerges, and from what specific neuronal process, is in debate. One possible explanation of consciousness, proposed here, is that it is a construct of the social perceptual machinery. Humans have specialized neuronal machinery that allows us to be socially intelligent. The primary role for this machinery is to construct models of other people's minds thereby gaining some ability to predict the behavior of other individuals. In the present hypothesis, awareness is a perceptual reconstruction of attentional state; and the machinery that computes information about other people's awareness is the same machinery that computes information about our own awareness. The present article brings together a variety of lines of evidence including experiments on the neural basis of social perception, on hemispatial neglect, on the out-of-body experience, on mirror neurons, and on the mechanisms of decision-making, to explore the possibility that awareness is a construct of the social machinery in the brain.

A framework for local cortical oscillation patterns

Oscillations are a pervasive feature of neuronal activity in the cerebral cortex. Here, we propose a framework for understanding local cortical oscillation patterns in cognition: two classes of network interactions underlying two classes of cognitive functions produce different local oscillation patterns. Local excitatory-inhibitory interactions shape neuronal representations of sensory, motor and cognitive variables, and produce local gamma-band oscillations. By contrast, the linkage of such representations by integrative functions such as decision-making is mediated by long-range cortical interactions, which yield more diverse local oscillation patterns often involving the beta range. This framework reconciles different cortical oscillation patterns observed in recent studies and helps to understand the link between cortical oscillations and the fMRI signal. Our framework highlights the notion that cortical oscillations index the specific circuit-level mechanisms of cognition.