Choosing the greater of two goods: neural currencies for valuation and decision making

Cognition and medial frontal cortex in health and disease

PURPOSE OF REVIEW

Recent work on the role of the medial frontal cortex in cognition and its involvement in neurological disorders is critically reviewed.

RECENT FINDINGS

The highly influential notion of conflict monitoring by the anterior cingulate has been called into question by monkey single-cell neurophysiology and lesion studies in monkeys and humans. An alternative role for this region in adapting behaviour in response to changing demands over time is gaining support. By contrast, the more dorsally placed presupplementary motor area and supplementary eye field have been implicated in direct executive control in situations of response conflict. Although more rostral medial areas have been linked to complex cognitive operations involving references to the self, conceptual obstacles make the evidence difficult to interpret. The role of the orbitofrontal cortex in guiding action based on value has been reinforced.

SUMMARY

The role of the medial frontal cortex in cognition continues to generate both interest and controversy. A few striking discrepancies between data from functional imaging and interventional techniques illustrate the hazards of drawing strong conclusions from merely correlative evidence. More broadly, a case can be made for tempering the empirical enthusiasm here with a little more theoretical restraint.

The role of ventromedial prefrontal cortex in decision making: judgment under uncertainty or judgment per se?

Ventromedial prefrontal cortex (VMF) is thought to be important in human decision making, but studies to date have focused on decision making under conditions of uncertainty, including risky or ambiguous decisions. Other lines of evidence suggest that this area of the brain represents quite basic information about the relative "economic" value of options, predicting a role for this region in value-based decision making even in the absence of uncertainty. We tested this prediction in human subjects with VMF damage. Preference judgment is a simple form of value-based decision making under certainty. We asked whether VMF damage in humans would lead to inconsistent preference judgments in a simple pairwise choice task. Twenty-one participants with focal damage to the frontal lobes were compared with 19 age- and education-matched control subjects. Subjects with VMF damage were significantly more inconsistent in their preferences than controls, whereas those with frontal damage that spared the VMF performed normally. These results argue that VMF plays a necessary role in certain as well as uncertain decision making in humans.

Neural mechanism for stochastic behaviour during a competitive game

Previous studies have shown that non-human primates can generate highly stochastic choice behaviour, especially when this is required during a competitive interaction with another agent. To understand the neural mechanism of such dynamic choice behaviour, we propose a biologically plausible model of decision making endowed with synaptic plasticity that follows a reward-dependent stochastic Hebbian learning rule. This model constitutes a biophysical implementation of reinforcement learning, and it reproduces salient features of behavioural data from an experiment with monkeys playing a matching pennies game. Due to interaction with an opponent and learning dynamics, the model generates quasi-random behaviour robustly in spite of intrinsic biases. Furthermore, non-random choice behaviour can also emerge when the model plays against a non-interactive opponent, as observed in the monkey experiment. Finally, when combined with a meta-learning algorithm, our model accounts for the slow drift in the animal's strategy based on a process of reward maximization.

The short-latency dopamine signal: a role in discovering novel actions?

An influential concept in contemporary computational neuroscience is the reward prediction error hypothesis of phasic dopaminergic function. It maintains that midbrain dopaminergic neurons signal the occurrence of unpredicted reward, which is used in appetitive learning to reinforce existing actions that most often lead to reward. However, the availability of limited afferent sensory processing and the precise timing of dopaminergic signals suggest that they might instead have a central role in identifying which aspects of context and behavioural output are crucial in causing unpredicted events.

The rodent orbitofrontal cortex gets time and direction

The orbitofrontal cortex (OFC) helps direct decision making through its flexible coding of reward and economic value. In this issue of Neuron, papers by Roesch et al. and Feierstein et al. demonstrate the importance of temporal and spatial features to processing in the rodent OFC.

Imaging valuation models in human choice

To make a decision, a system must assign value to each of its available choices. In the human brain, one approach to studying valuation has used rewarding stimuli to map out brain responses by varying the dimension or importance of the rewards. However, theoretical models have taught us that value computations are complex, and so reward probes alone can give only partial information about neural responses related to valuation. In recent years, computationally principled models of value learning have been used in conjunction with noninvasive neuroimaging to tease out neural valuation responses related to reward-learning and decision-making. We restrict our review to the role of these models in a new generation of experiments that seeks to build on a now-large body of diverse reward-related brain responses. We show that the models and the measurements based on them point the way forward in two important directions: the valuation of time and the valuation of fictive experience.

Linear-Nonlinear-Poisson models of primate choice dynamics

The equilibrium phenomenon of matching behavior traditionally has been studied in stationary environments. Here we attempt to uncover the local mechanism of choice that gives rise to matching by studying behavior in a highly dynamic foraging environment. In our experiments, 2 rhesus monkeys (Macacca mulatta) foraged for juice rewards by making eye movements to one of two colored icons presented on a computer monitor, each rewarded on dynamic variable-interval schedules. Using a generalization of Wiener kernel analysis, we recover a compact mechanistic description of the impact of past reward on future choice in the form of a Linear-Nonlinear-Poisson model. We validate this model through rigorous predictive and generative testing. Compared to our earlier work with this same data set, this model proves to be a better description of choice behavior and is more tightly correlated with putative neural value signals. Refinements over previous models include hyperbolic (as opposed to exponential) temporal discounting of past rewards, and differential (as opposed to fractional) comparisons of option value. Through numerical simulation we find that within this class of strategies, the model parameters employed by animals are very close to those that maximize reward harvesting efficiency.

The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans

Many real-life decision-making problems incorporate higher-order structure, involving interdependencies between different stimuli, actions, and subsequent rewards. It is not known whether brain regions implicated in decision making, such as the ventromedial prefrontal cortex (vmPFC), use a stored model of the task structure to guide choice (model-based decision making) or merely learn action or state values without assuming higher-order structure as in standard reinforcement learning. To discriminate between these possibilities, we scanned human subjects with functional magnetic resonance imaging while they performed a simple decision-making task with higher-order structure, probabilistic reversal learning. We found that neural activity in a key decision-making region, the vmPFC, was more consistent with a computational model that exploits higher-order structure than with simple reinforcement learning. These results suggest that brain regions, such as the vmPFC, use an abstract model of task structure to guide behavioral choice, computations that may underlie the human capacity for complex social interactions and abstract strategizing.

IL-6 knockout mice exhibit resistance to stress-induced development of depression-like behaviors

Cytokine-dependent mechanisms in the CNS have been implicated in the pathogenesis of depression. Interleukin-6 is upregulated in depressed patients and dowregulated by antidepressants. It is, however, unknown whether IL-6 is involved in the pathogenesis of depression. We subjected IL-6-deficient mice (IL-6(-/-)) to depression-related tests (learned helplessness, forced swimming, tail suspension, sucrose preference). We also investigated IL-6 in the hippocampus of stressed wild-type mice. IL-6(-/-) mice showed reduced despair in the forced swim, and tail suspension test, and enhanced hedonic behavior. Moreover, IL-6(-/-) mice exhibited resistance to helplessness. This resistance may be caused by the lack of IL-6, because stress increased IL-6 expression in wild-type hippocampi. This suggests that IL-6 is a component in molecular mechanisms in the pathogenesis of depression. IL-6(-/-) mice represent tools to study IL-6-dependent signaling pathways in the pathophysiology of depression in vivo. Moreover, these mice may support the screening of compounds for depression by altering cytokine-mediated signaling.

Resolution of Uncertainty in Prefrontal Cortex

Making optimal decisions in the face of uncertain or incomplete information arises as a common problem in everyday behavior, but the neural processes underlying this ability remain poorly understood. A typical case is navigation, in which a subject has to search for a known goal from an unknown location. Navigating under uncertain conditions requires making decisions on the basis of the current belief about location and updating that belief based on incoming information. Here, we use functional magnetic resonance imaging during a maze navigation task to study neural activity relating to the resolution of uncertainty as subjects make sequential decisions to reach a goal. We show that distinct regions of prefrontal cortex are engaged in specific computational functions that are well described by a Bayesian model of decision making. This permits efficient goal-oriented navigation and provides new insights into decision making by humans.

A biophysically based neural model of matching law behavior: melioration by stochastic synapses

In experiments designed to uncover the neural basis of adaptive decision making in a foraging environment, neuroscientists have reported single-cell activities in the lateral intraparietal cortex (LIP) that are correlated with choice options and their subjective values. To investigate the underlying synaptic mechanism, we considered a spiking neuron model of decision making endowed with synaptic plasticity that follows a reward-dependent stochastic Hebbian learning rule. This general model is tested in a matching task in which rewards on two targets are scheduled randomly with different rates. Our main results are threefold. First, we show that plastic synapses provide a natural way to integrate past rewards and estimate the local (in time) "return" of a choice. Second, our model reproduces the matching behavior (i.e., the proportional allocation of choices matches the relative reinforcement obtained on those choices, which is achieved through melioration in individual trials). Our model also explains the observed "undermatching" phenomenon and points to biophysical constraints (such as finite learning rate and stochastic neuronal firing) that set the limits to matching behavior. Third, although our decision model is an attractor network exhibiting winner-take-all competition, it captures graded neural spiking activities observed in LIP, when the latter were sorted according to the choices and the difference in the returns for the two targets. These results suggest that neurons in LIP are involved in selecting the oculomotor responses, whereas rewards are integrated and stored elsewhere, possibly by plastic synapses and in the form of the return rather than income of choice options.

A recurrent network mechanism of time integration in perceptual decisions

Recent physiological studies using behaving monkeys revealed that, in a two-alternative forced-choice visual motion discrimination task, reaction time was correlated with ramping of spike activity of lateral intraparietal cortical neurons. The ramping activity appears to reflect temporal accumulation, on a timescale of hundreds of milliseconds, of sensory evidence before a decision is reached. To elucidate the cellular and circuit basis of such integration times, we developed and investigated a simplified two-variable version of a biophysically realistic cortical network model of decision making. In this model, slow time integration can be achieved robustly if excitatory reverberation is primarily mediated by NMDA receptors; our model with only fast AMPA receptors at recurrent synapses produces decision times that are not comparable with experimental observations. Moreover, we found two distinct modes of network behavior, in which decision computation by winner-take-all competition is instantiated with or without attractor states for working memory. Decision process is closely linked to the local dynamics, in the "decision space" of the system, in the vicinity of an unstable saddle steady state that separates the basins of attraction for the two alternative choices. This picture provides a rigorous and quantitative explanation for the dependence of performance and response time on the degree of task difficulty, and the reason for which reaction times are longer in error trials than in correct trials as observed in the monkey experiment. Our reduced two-variable neural model offers a simple yet biophysically plausible framework for studying perceptual decision making in general.

From crawling to cognition: analyzing the dynamical interactions among populations of neurons

By using multi-electrode arrays or optical imaging, investigators can now record from many individual neurons in various parts of nervous systems simultaneously while an animal performs sensory, motor or cognitive tasks. Given the large multidimensional datasets that are now routinely generated, it is often not obvious how to find meaningful results within the data. The analysis of neuronal-population recordings typically involves two steps: the extraction of relevant dynamics from neural data, and then use of the dynamics to classify and discriminate features of a stimulus or behavior. We focus on the application of techniques that emphasize interactions among the recorded neurons rather than using just the correlations between individual neurons and a perception or a behavior. An understanding of modern analysis techniques is crucially important for researchers interested in the co-varying activity among populations of neurons or even brain regions.

Behavioural assessment of drug reinforcement and addictive features in rodents: an overview

Some psychoactive drugs are abused because of their ability to act as reinforcers. As a consequence behavioural patterns (such as drug-seeking/drug-taking behaviours) are promoted that ensure further drug consumption. After prolonged drug self-administration, some individuals lose control over their behaviour so that these drug-seeking/taking behaviours become compulsive, pervading almost all life activities and precipitating the loss of social compatibility. Thus, the syndrome of addictive behaviour is qualitatively different from controlled drug consumption. Drug-induced reinforcement can be assessed directly in laboratory animals by either operant or non-operant self-administration methods, by classical conditioning-based paradigms such as conditioned place preference or sign tracking, by facilitation of intracranial electric self-stimulation, or, alternatively by drug-induced memory enhancement. In contrast, addiction cannot be modelled in animals, at least as a whole, within the constraints of the laboratory. However, various procedures have been proposed as possible rodent analogues of addiction's major elements including compulsive drug seeking, relapse, loss of control/impulsivity, and continued drug consumption despite negative consequences. This review provides an extensive overview and a critical evaluation of the methods currently used for studying drug-induced reinforcement as well as specific features of addictive behaviour. In addition, comic strips that illustrate behavioural methods used in the drug abuse field are provided given for free download under http://www.zi-mannheim/psychopharmacology.de.

Population approach to a neural discrimination task

This article gives insights into the possible neuronal processes involved in visual discrimination. We study the performance of a spiking network of Integrate-and-Fire (IF) neurons when performing a benchmark discrimination task. The task we adopted consists of determining the direction of moving dots in a noisy context using similar stimuli to those in the experiments of Newsome and colleagues. We present a neural model that performs the discrimination involved in this task. By varying the synaptic parameters of the IF neurons, we illustrate the counter-intuitive importance of the second-order statistics (input noise) in improving the discrimination accuracy of the model. We show that measuring the Firing Rate (FR) over a population enables the model to discriminate in realistic times, and even surprisingly significantly increases its discrimination accuracy over the single neuron case, despite the faster processing. We also show that increasing the input noise increases the discrimination accuracy but only at the expense of the speed at which we can read out the FR.

Basal ganglia orient eyes to reward

Expectation of reward motivates our behaviors and influences our decisions. Indeed, neuronal activity in many brain areas is modulated by expected reward. However, it is still unclear where and how the reward-dependent modulation of neuronal activity occurs and how the reward-modulated signal is transformed into motor outputs. Recent studies suggest an important role of the basal ganglia. Sensorimotor/cognitive activities of neurons in the basal ganglia are strongly modulated by expected reward. Through their abundant outputs to the brain stem motor areas and the thalamocortical circuits, the basal ganglia appear capable of producing body movements based on expected reward. A good behavioral measure to test this hypothesis is saccadic eye movement because its brain stem mechanism has been extensively studied. Studies from our laboratory suggest that the basal ganglia play a key role in guiding the gaze to the location where reward is available. Neurons in the caudate nucleus and the substantia nigra pars reticulata are extremely sensitive to the positional difference in expected reward, which leads to a bias in excitability between the superior colliculi such that the saccade to the to-be-rewarded position occurs more quickly. It is suggested that the reward modulation occurs in the caudate where cortical inputs carrying spatial signals and dopaminergic inputs carrying reward-related signals are integrated. These data support a specific form of reinforcement learning theories, but also suggest further refinement of the theory.

Deciding how to decide: ventromedial frontal lobe damage affects information acquisition in multi-attribute decision making

Ventromedial frontal lobe (VMF) damage is associated with impaired decision making. Recent efforts to understand the functions of this brain region have focused on its role in tracking reward, punishment and risk. However, decision making is complex, and frontal lobe damage might be expected to affect it at other levels. This study used process-tracing techniques to explore the effect of VMF damage on multi-attribute decision making under certainty. Thirteen subjects with focal VMF damage were compared with 11 subjects with frontal damage that spared the VMF and 21 demographically matched healthy control subjects. Participants chose rental apartments in a standard information board task drawn from the literature on normal decision making. VMF subjects performed the decision making task in a way that differed markedly from all other groups, favouring an 'alternative-based' information acquisition strategy (i.e. they organized their information search around individual apartments). In contrast, both healthy control subjects and subjects with damage predominantly involving dorsal and/or lateral prefrontal cortex pursued primarily 'attribute-based' search strategies (in which information was acquired about categories such as rent and noise level across several apartments). This difference in the pattern of information acquisition argues for systematic differences in the underlying decision heuristics and strategies employed by subjects with VMF damage, which in turn may affect the quality of their choices. These findings suggest that the processes supported by ventral and medial prefrontal cortex need to be conceptualized more broadly, to account for changes in decision making under conditions of certainty, as well as uncertainty, following damage to these areas.

Action control in visual neglect

Patients with unilateral neglect show a variety of impairments when reaching towards objects in contralesional space. The basis of these deficits could be perceptual, motor or at one of the intermediate stages linking these processes. Here, we review studies of visually guided reaching in neglect and integrate these results with findings from normal human and monkey action control. We consider evidence which shows that neglect patients can be slow to initiate or execute reaches particularly to a contralesional target. We discuss the directional and spatial deficits that may interact to contribute to such reaching abnormalities and highlight the importance of effective target selection and on-line guidance, exploring the idea that deficits in these mechanisms underlie increased susceptibility to ipsilesional visual distraction in neglect. We also examine the relationship between optic ataxia and neglect by considering two illustrative cases, one with pure optic ataxia and the other with optic ataxia plus neglect, which reveal differences in the anatomical substrates of the two syndromes. We conclude that many patients with neglect make abnormal visually guided reaches, but the pattern of reaching deficits is highly variable, most likely reflecting heterogeneity of lesion location across subjects. Rather than being specific to the neglect syndrome, abnormalities of reaching in these patients may correspond to the extent of damage to the visuomotor control system which involves critical regions in both the parietal and frontal cortex, the white matter tracts connecting them and subcortical regions. Thus, the action control deficits in neglect may be conceptualised as a range of impairments affecting multiple stages in the visuomotor control process.

How brains beware: neural mechanisms of emotional attention

Emotional processes not only serve to record the value of sensory events, but also to elicit adaptive responses and modify perception. Recent research using functional brain imaging in human subjects has begun to reveal neural substrates by which sensory processing and attention can be modulated by the affective significance of stimuli. The amygdala plays a crucial role in providing both direct and indirect top-down signals on sensory pathways, which can influence the representation of emotional events, especially when related to threat. These modulatory effects implement specialized mechanisms of 'emotional attention' that might supplement but also compete with other sources of top-down control on perception. This work should help to elucidate the neural processes and temporal dynamics governing the integration of cognitive and affective influences in attention and behaviour.

Beetles, boxes and brain cells: neural mechanisms underlying valuation and learning

Sensory cues in the environment can predict the availability of reward. Through experience, humans and animals learn these predictions and use them to guide their actions. For example, we can learn to discriminate chanterelles from ordinary champignons through experience. Assuming the development of a taste for the complex and lingering flavors of chanterelles, we therefore learn to value the same action--picking mushrooms--differentially depending upon the appearance of a mushroom. One major goal of cognitive neuroscience is to understand the neural mechanisms that underlie this sort of learning. Because the acquisition of rewards motivates much behavior, recent efforts have focused on describing the neural signals related to learning the value of stimuli and actions. Neurons in the basal ganglia, in midbrain dopamine areas, in frontal and parietal cortices and in other brain areas, all modulate their activity in relation to aspects of learning. By training monkeys on various behavioral tasks, recent studies have begun to characterize how neural signals represent distinct processes, such as the timing of events, motivation, absolute (objective) and relative (subjective) valuation, and the formation of associative links between stimuli and potential actions. In addition, a number of studies have either further characterized dopamine signals or sought to determine how such signaling might interact with target structures, such as the striatum and rhinal cortex, to underlie learning.