Hold your horses: A dynamic computational role for the subthalamic nucleus in decision making

Impulsivity and Parkinson's disease: more than just disinhibition

In the last few years it has become clear that impulsivity is a complex behaviour composed of different domains and dependent on different neural networks. The proposed pathogenetic mechanisms for the emergence of impulsivity disorders in Parkinson's Disease (PD) can be broadly separated into three potentially interacting processes: the contribution of premorbid susceptibility to impulsivity, the contribution of the disease itself to the behaviour and the potential contribution of therapeutic agents. Growing evidence suggests that dopamine and the subthalamic nucleus are playing a certain role in the pathophysiology of different aspects of impulsivity. In this review, we summarise the main concepts defining various components of impulsivity both in healthy subjects and patients affected by PD.

Neural basis of adaptive response time adjustment during saccade countermanding

Humans and macaque monkeys adjust their response time adaptively in stop-signal (countermanding) tasks, responding slower after stop-signal trials than after control trials with no stop signal. We investigated the neural mechanism underlying this adaptive response time adjustment in macaque monkeys performing a saccade countermanding task. Earlier research showed that movements are initiated when the random accumulation of presaccadic movement-related activity reaches a fixed threshold. We found that a systematic delay in response time after stop-signal trials was accomplished not through a change of threshold, baseline, or accumulation rate, but instead through a change in the time when activity first began to accumulate. The neurons underlying movement initiation have been identified with stochastic accumulator models of response time performance. Therefore, this new result provides surprising new insights into the neural instantiation of stochastic accumulator models and the mechanisms through which executive control can be exerted.

Stopping a response has global or nonglobal effects on the motor system depending on preparation

Much research has focused on how people stop initiated response tendencies when instructed by a signal. Stopping of this kind appears to have global effects on the motor system. For example, by delivering transcranial magnetic stimulation (TMS) over the leg area of the primary motor cortex, it is possible to detect suppression in the leg when the hand is being stopped (Badry R et al. Suppression of human cortico-motoneuronal excitability during the stop-signal task. Clin Neurophysiol 120: 1717-1723, 2009). Here, we asked if such "global suppression" can be observed proactively, i.e., when people anticipate they might have to stop. We used a conditional stop signal task, which allows the measurement of both an "anticipation phase" (i.e., where proactive control is applied) and a "stopping" phase. TMS was delivered during the anticipation phase (experiment 1) and also during the stopping phase (experiments 1 and 2) to measure leg excitability. During the anticipation phase, we did not observe leg suppression, but we did during the stopping phase, consistent with Badry et al. (2009). Moreover, when we split the subject groups into those who slowed down behaviorally (i.e., exercised proactive control) and those who did not, we found that subjects who slowed did not show leg suppression when they stopped, whereas those who did not slow did show leg suppression when they stopped. These results suggest that if subjects prepare to stop, then they do so without global effects on the motor system. Thus, preparation allows them to stop more selectively.

Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold

It takes effort and time to tame one's impulses. Although medial prefrontal cortex (mPFC) is broadly implicated in effortful control over behavior, the subthalamic nucleus (STN) is specifically thought to contribute by acting as a brake on cortico-striatal function during decision conflict, buying time until the right decision can be made. Using the drift diffusion model of decision making, we found that trial-to-trial increases in mPFC activity (EEG theta power, 4-8 Hz) were related to an increased threshold for evidence accumulation (decision threshold) as a function of conflict. Deep brain stimulation of the STN in individuals with Parkinson's disease reversed this relationship, resulting in impulsive choice. In addition, intracranial recordings of the STN area revealed increased activity (2.5-5 Hz) during these same high-conflict decisions. Activity in these slow frequency bands may reflect a neural substrate for cortico-basal ganglia communication regulating decision processes.

Deep brain stimulation of the subthalamic nucleus transiently enhances loss-chasing behaviour in patients with Parkinson's disease

Dopaminergic treatments are associated with impulse control disorders such as pathological gambling in a subset of patients with Parkinson's Disease. While deep brain stimulation of the subthalamic nucleus has been reported to reduce symptoms of impulse control disorders in some Parkinson's Disease patients, little is known about its specific effects on gambling behaviour. In this experiment, we investigated the effects of deep brain stimulation of the subthalamic nucleus on one of the central features of pathological gambling: the tendency to chase losses. Loss-chasing is associated with impaired control over gambling behaviour and it is one of the most salient features of pathological gambling as it presents in the clinic. Twenty two patients with advanced idiopathic Parkinson's Disease and chronically implanted subthalamic nucleus electrodes for deep brain stimulation completed a simple laboratory model of loss-chasing behaviour twice: once with and once without stimulation. Exploratory analysis indicated that deep brain stimulation of the subthalamic nucleus increased the value of losses chased by patients with Parkinson's Disease when shifting from off- to on-stimulation. These effects were not attributable to changes in state affect or to the motor impairments produced by the withdrawal of deep brain stimulation of the subthalamic nucleus. The effects of the stimulation on the value of losses chased were more pronounced in female than in male patients and reduced in patients taking dopamine receptor agonists. Collectively, these results suggest that deep brain stimulation of the subthalamic nucleus can transiently alter the evaluation of accumulated losses during gambling episodes in idiopathic Parkinson's Disease.

The Neurodynamics of Cognition: A Tutorial on Computational Cognitive Neuroscience

Computational Cognitive Neuroscience (CCN) is a new field that lies at the intersection of computational neuroscience, machine learning, and neural network theory (i.e., connectionism). The ideal CCN model should not make any assumptions that are known to contradict the current neuroscience literature and at the same time provide good accounts of behavior and at least some neuroscience data (e.g., single-neuron activity, fMRI data). Furthermore, once set, the architecture of the CCN network and the models of each individual unit should remain fixed throughout all applications. Because of the greater weight they place on biological accuracy, CCN models differ substantially from traditional neural network models in how each individual unit is modeled, how learning is modeled, and how behavior is generated from the network. A variety of CCN solutions to these three problems are described. A real example of this approach is described, and some advantages and limitations of the CCN approach are discussed.

The Basal Ganglia as a Substrate for the Multiple Actions of Amphetamines

Amphetamines are psychostimulant drugs with high abuse potential. Acute and chronic doses of amphetamines affect dopamine (DA) neurotransmission in the basal ganglia. The basal ganglia are a group of subcortical nuclei that are anatomically positioned to integrate cognitive, motor and sensorimotor inputs from the cortex. Amphetamines can differentially alter the functioning of specific BG circuits to produce neurochemical changes that affect cognition, movement, and drug seeking behavior through their effects on DA neurotransmission. This review focuses on how alterations in dopaminergic neurotransmission within distinct basal ganglia pathways can modify their functional output to predict and explain the acute and long term behavioral consequences of amphetamine exposure.

Dopaminergic genes predict individual differences in susceptibility to confirmation bias

The striatum is critical for the incremental learning of values associated with behavioral actions. The prefrontal cortex (PFC) represents abstract rules and explicit contingencies to support rapid behavioral adaptation in the absence of cumulative experience. Here we test two alternative models of the interaction between these systems, and individual differences thereof, when human subjects are instructed with prior information about reward contingencies that may or may not be accurate. Behaviorally, subjects are overly influenced by prior instructions, at the expense of learning true reinforcement statistics. Computational analysis found that this pattern of data is best accounted for by a confirmation bias mechanism in which prior beliefs--putatively represented in PFC--influence the learning that occurs in the striatum such that reinforcement statistics are distorted. We assessed genetic variants affecting prefrontal and striatal dopaminergic neurotransmission. A polymorphism in the COMT gene (rs4680), associated with prefrontal dopaminergic function, was predictive of the degree to which participants persisted in responding in accordance with prior instructions even as evidence against their veracity accumulated. Polymorphisms in genes associated with striatal dopamine function (DARPP-32, rs907094, and DRD2, rs6277) were predictive of learning from positive and negative outcomes. Notably, these same variants were predictive of the degree to which such learning was overly inflated or neglected when outcomes are consistent or inconsistent with prior instructions. These findings indicate dissociable neurocomputational and genetic mechanisms by which initial biases are strengthened by experience.

Dopamine and effort-based decision making

Motivational theories of choice focus on the influence of goal values and strength of reinforcement to explain behavior. By contrast relatively little is known concerning how the cost of an action, such as effort expended, contributes to a decision to act. Effort-based decision making addresses how we make an action choice based on an integration of action and goal values. Here we review behavioral and neurobiological data regarding the representation of effort as action cost, and how this impacts on decision making. Although organisms expend effort to obtain a desired reward there is a striking sensitivity to the amount of effort required, such that the net preference for an action decreases as effort cost increases. We discuss the contribution of the neurotransmitter dopamine (DA) toward overcoming response costs and in enhancing an animal's motivation toward effortful actions. We also consider the contribution of brain structures, including the basal ganglia and anterior cingulate cortex, in the internal generation of action involving a translation of reward expectation into effortful action.

Prefrontal cortex and impulsive decision making

Impulsivity refers to a set of heterogeneous behaviors that are tuned suboptimally along certain temporal dimensions. Impulsive intertemporal choice refers to the tendency to forego a large but delayed reward and to seek an inferior but more immediate reward, whereas impulsive motor responses also result when the subjects fail to suppress inappropriate automatic behaviors. In addition, impulsive actions can be produced when too much emphasis is placed on speed rather than accuracy in a wide range of behaviors, including perceptual decision making. Despite this heterogeneous nature, the prefrontal cortex and its connected areas, such as the basal ganglia, play an important role in gating impulsive actions in a variety of behavioral tasks. Here, we describe key features of computations necessary for optimal decision making and how their failures can lead to impulsive behaviors. We also review the recent findings from neuroimaging and single-neuron recording studies on the neural mechanisms related to impulsive behaviors. Converging approaches in economics, psychology, and neuroscience provide a unique vista for better understanding the nature of behavioral impairments associated with impulsivity.

From reactive to proactive and selective control: developing a richer model for stopping inappropriate responses

A better understanding of the neural systems underlying impulse control is important for psychiatry. Although most impulses are motivational or emotional rather than motoric per se, it is research into the neural architecture of motor response control that has made the greatest strides. This article reviews recent developments in the cognitive neuroscience of stopping responses. Most research of this kind has focused on reactive control-that is, how subjects stop a response outright when instructed by a signal. It is argued that reactive paradigms are limited as models of control relevant to psychiatry. Instead, a set of paradigms is advocated that begins to model proactive inhibitory control-that is, how a subject prepares to stop an upcoming response tendency. Proactive inhibitory control is generated according to the goals of the subject rather than by an external signal, and it can be selectively targeted at a particular response tendency. This may have wider validity than reactive control as an experimental model for stopping inappropriate responses.

The Subthalamic Nucleus becomes a Generator of Bursts in the Dopamine-Depleted State. Its High Frequency Stimulation Dramatically Weakens Transmission to the Globus Pallidus

Excessive burst firing in the dopamine-depleted basal ganglia correlates with severe motor symptoms of Parkinson's disease that are attenuated by high frequency electrical stimulation of the subthalamic nucleus (STN). Here we test the hypothesis that pathological bursts in dopamine-deprived basal ganglia are generated within the STN and transmitted to globus pallidus neurons. To answer this question we recorded excitatory synaptic currents and potentials from subthalamic and pallidal neurons in the basal ganglia slice (BGS) from dopamine-depleted mice while continuously blocking GABA_A receptors. In control mice, a single electrical stimulus delivered to the internal capsule or the rostral pole of the STN evoked a short duration, small amplitude, monosynaptic EPSC in subthalamic neurons. In contrast, in the dopamine-depleted BGS, this monosynaptic EPSC was amplified and followed by a burst of polysynaptic EPSCs that eventually reverberated three to seven times, providing a long lasting response that gave rise to bursts of EPSCs and spikes in GP neurons. Repetitive (10–120 Hz) stimulation delivered to the STN in the dopamine-depleted BGS attenuated STN-evoked bursts of EPSCs in pallidal neurons after several minutes of stimulation but only high frequency (90–120 Hz) stimulation replaced them with small amplitude EPSCs at 20 Hz. We propose that the polysynaptic pathway within the STN amplifies subthalamic responses to incoming excitation in the dopamine-depleted basal ganglia, thereby transforming the STN into a burst generator and entraining pallidal neurons in pathogenic bursting activities. High frequency stimulation of the STN prevents the transmission of this pathological activity to globus pallidus and imposes a new glutamatergic synaptic noise on pallidal neurons.

Neural mechanisms of saccade target selection: gated accumulator model of the visual-motor cascade

We review a new computational model developed to understand how evidence about stimulus salience in visual search is translated into a saccade command. The model uses the activity of visually responsive neurons in the frontal eye field as evidence for stimulus salience that is accumulated in a network of stochastic accumulators to produce accurate and timely saccades. We discovered that only when the input to the accumulation process was gated could the model account for the variability in search performance and predict the dynamics of movement neuron discharge rates. This union of cognitive modeling and neurophysiology indicates how the visual-motor transformation can occur, and provides a concrete mapping between neuron function and specific cognitive processes.

Modulation of short intra-cortical inhibition during action reprogramming

Actions are selected in the context of environmental demands and internal goals. Since both change continuously it is often necessary to inhibit a prepared action plan in favour of an alternative, a process we refer to as action reprogramming. Previous studies have established that a frontal/basal ganglia network exerts top-down control over the primary motor cortex (M1) during action reprogramming. The current study focuses on the role of M1 itself during action reprogramming. Participants were asked to perform a behavioural task that required them to either execute a prepared response or to reprogram an alternative response. Paired-pulse TMS was used to investigate short-interval intra-cortical inhibition (SICI) during these action execution and action reprogramming trials. Normal action execution was associated with sustained SICI in the M1 during both trials in which the contralateral hand was to respond and trials in which the ipsilateral hand was to respond. In contrast, reprogramming towards an alternative action was associated with a progressive release of SICI in M1 involved in the execution of the novel response. This release started 125 ms after the cue telling the participants to reprogram their action. This time point is consistent with previous results showing a facilitatory influence of the pre-supplementary motor area (pre-SMA) on the M1 at the same delay. Hence, SICI might be a potential candidate mechanism through which frontal lobe areas could influence primary motor cortex output.

Motor learning: spare the rod to benefit the child?

A new study has found that individuals who were rewarded while they learned a motor task performed it much better one month later than those who were punished or received nothing. Long-term memories depend on events experienced during learning.

Computational models of motivated action selection in corticostriatal circuits

Computational models of the basal ganglia have matured and received increasing attention over the last decade. This article reviews some of the theoretical advances offered by these models, focusing on motor and cognitive action selection, learning, and the interaction between multiple corticostriatal circuits in selection and learning.

Dorsolateral prefrontal cortex modulates striatal reward encoding during reappraisal of reward anticipation

Recent research showed that cognitive emotion regulation (ER) both increases activity in the dorsolateral prefrontal cortex (DLPFC) and decreases striatal responsivity to monetary rewards. Using a mixed monetary incentive delay/memory task as well as functional magnetic resonance imaging, we tested in healthy subjects whether ER effectively attenuates striatal reward encoding during the anticipation of reward (€1.00 vs. €0.05 reward cues) as well as subsequent target reaction times (RTs), which are an indicator of motivation to obtain reward. ER significantly diminished feelings of pleasant anticipation and slowed down €1.00 target RT. At the neural level, ER increased activity in the DLPFC and attenuated reward encoding in the left putamen. Analyses of psychophysiological interaction revealed that DLPFC activity correlated more positively with putamen activity during €0.05 than during €1.00 reward trials. Furthermore, parametric modulations showed that anticipatory left putamen activity correlated with target RT during nonregulation. No such correlation could be observed during ER, suggesting that ER had abolished preparatory target RT encoding. Our results provide evidence that ER can attenuate behavioral and striatal measures of reward-related motivation and motor preparation. Furthermore, the present findings suggest that the DLPFC might contribute to successful regulation of reward via increased promotion of low-reward responses.

The arbitration-extension hypothesis: a hierarchical interpretation of the functional organization of the Basal Ganglia

Based on known anatomy and physiology, we present a hypothesis where the basal ganglia motor loop is hierarchically organized in two main subsystems: the arbitration system and the extension system. The arbitration system, comprised of the subthalamic nucleus, globus pallidus, and pedunculopontine nucleus, serves the role of selecting one out of several candidate actions as they are ascending from various brain stem motor regions and aggregated in the centromedian thalamus or descending from the extension system or from the cerebral cortex. This system is an action-input/action-output system whose winner-take-all mechanism finds the strongest response among several candidates to execute. This decision is communicated back to the brain stem by facilitating the desired action via cholinergic/glutamatergic projections and suppressing conflicting alternatives via GABAergic connections. The extension system, comprised of the striatum and, again, globus pallidus, can extend the repertoire of responses by learning to associate novel complex states to certain actions. This system is a state-input/action-output system, whose organization enables it to encode arbitrarily complex Boolean logic rules using striatal neurons that only fire given specific constellations of inputs (Boolean AND) and pallidal neurons that are silenced by any striatal input (Boolean OR). We demonstrate the capabilities of this hierarchical system by a computational model where a simulated generic “animal” interacts with an environment by selecting direction of movement based on combinations of sensory stimuli, some being appetitive, others aversive or neutral. While the arbitration system can autonomously handle conflicting actions proposed by brain stem motor nuclei, the extension system is required to execute learned actions not suggested by external motor centers. Being precise in the functional role of each component of the system, this hypothesis generates several readily testable predictions.

From reinforcement learning models to psychiatric and neurological disorders

Over the last decade and a half, reinforcement learning models have fostered an increasingly sophisticated understanding of the functions of dopamine and cortico-basal ganglia-thalamo-cortical (CBGTC) circuits. More recently, these models, and the insights that they afford, have started to be used to understand important aspects of several psychiatric and neurological disorders that involve disturbances of the dopaminergic system and CBGTC circuits. We review this approach and its existing and potential applications to Parkinson's disease, Tourette's syndrome, attention-deficit/hyperactivity disorder, addiction, schizophrenia and preclinical animal models used to screen new antipsychotic drugs. The approach's proven explanatory and predictive power bodes well for the continued growth of computational psychiatry and computational neurology.

Impulse control and related disorders in Parkinson's disease patients treated with bilateral subthalamic nucleus stimulation: a review

Recently, impulse control and related disorders including punding and the dopamine dysregulation syndrome (DDS) have been increasingly recognized in treated patients with Parkinson's disease (PD). Especially the impulse control disorders (ICD) such as pathological gambling, hypersexuality, compulsive eating and buying may have dramatic repercussions on family, personal and professional life. Drug replacement therapy (DRT) is believed to play an important role in the onset of these behavioral disturbances. Although deep brain stimulation (DBS) of the subthalamic nucleus (STN) might be a therapeutic option for those patients with DRT-related behavior, it may also induce ICD. So far, little is known about the relationship between STN DBS and impulse control and related disorders. Our aim was to review the current knowledge on this relationship in PD patients. The available studies showed that stimulation of the STN is associated with both favorable and negative outcome in terms of impulse control and related disorders. Preoperative disorders may resolve or improve after STN DBS, but these can also worsen or show no change at all. Moreover, STN DBS can also reveal or even induce ICD. Possible explanations for this variability are proposed and suggestions for clinical management are given.

Deep Brain Stimulation and the Search for Identity

Ethical evaluation of deep brain stimulation as a treatment for Parkinson’s disease is complicated by results that can be described as involving changes in the patient’s identity. The risk of becoming another person following surgery is alarming for patients, caregivers and clinicians alike. It is one of the most urgent conceptual and ethical problems facing deep brain stimulation in Parkinson’s disease at this time. In our paper we take issue with this problem on two accounts. First, we elucidate what is meant by “becoming another person” from a conceptual point of view. After critically discussing two broad approaches we concentrate on the notion of “individual identity” which centers on the idea of “core attitudes”. Subsequently we discuss several approaches to determine what distinguishes core attitudes from those that are more peripheral. We argue for a “foundational-function model” highlighting the importance of specific dependency relations between these attitudes. Our second aim is to comment on the possibility to empirically measure changes in individual identity and argue that many of the instruments now commonly used in selecting and monitoring DBS-patients are inappropriate for this purpose. Future research in this area is advised combining a conceptual and an empirical approach as a basis of sound ethical appraisal.

Substantia nigra activity level predicts trial-to-trial adjustments in cognitive control

Effective adaptation to the demands of a changing environment requires flexible cognitive control. The medial and the lateral frontal cortices are involved in such control processes, putatively in close interplay with the BG. In particular, dopaminergic projections from the midbrain (i.e., from the substantia nigra [SN] and the ventral tegmental area) have been proposed to play a pivotal role in modulating the activity in these areas for cognitive control purposes. In that dopaminergic involvement has been strongly implicated in reinforcement learning, these ideas suggest functional links between reinforcement learning, where the outcome of actions shapes behavior over time, and cognitive control in a more general context, where no direct reward is involved. Here, we provide evidence from functional MRI in humans that activity in the SN predicts systematic subsequent trial-to-trial RT prolongations that are thought to reflect cognitive control in a stop-signal paradigm. In particular, variations in the activity level of the SN in one trial predicted the degree of RT prolongation on the subsequent trial, consistent with a modulating output signal from the SN being involved in enhancing cognitive control. This link between SN activity and subsequent behavioral adjustments lends support to theoretical accounts that propose dopaminergic control signals that shape behavior both in the presence and in the absence of direct reward. This SN-based modulatory mechanism is presumably mediated via a wider network that determines response speed in this task, including frontal and parietal control regions, along with the BG and the associated subthalamic nucleus.

Integration of reinforcement learning and optimal decision-making theories of the basal ganglia

This article seeks to integrate two sets of theories describing action selection in the basal ganglia: reinforcement learning theories describing learning which actions to select to maximize reward and decision-making theories proposing that the basal ganglia selects actions on the basis of sensory evidence accumulated in the cortex. In particular, we present a model that integrates the actor-critic model of reinforcement learning and a model assuming that the cortico-basal-ganglia circuit implements a statistically optimal decision-making procedure. The values of cortico-striatal weights required for optimal decision making in our model differ from those provided by standard reinforcement learning models. Nevertheless, we show that an actor-critic model converges to the weights required for optimal decision making when biologically realistic limits on synaptic weights are introduced. We also describe the model's predictions concerning reaction times and neural responses during learning, and we discuss directions required for further integration of reinforcement learning and optimal decision-making theories.

Opponency revisited: competition and cooperation between dopamine and serotonin

Affective valence lies on a spectrum ranging from punishment to reward. The coding of such spectra in the brain almost always involves opponency between pairs of systems or structures. There is ample evidence for the role of dopamine in the appetitive half of this spectrum, but little agreement about the existence, nature, or role of putative aversive opponents such as serotonin. In this review, we consider the structure of opponency in terms of previous biases about the nature of the decision problems that animals face, the conflicts that may thus arise between Pavlovian and instrumental responses, and an additional spectrum joining invigoration to inhibition. We use this analysis to shed light on aspects of the role of serotonin and its interactions with dopamine.

Serotonin and dopamine play complementary roles in gambling to recover losses

Continued gambling to recover losses--'loss chasing'--is a prominent feature of social and pathological gambling. However, little is known about the neuromodulators that influence this behavior. In three separate experiments, we investigated the role of serotonin activity, D(2)/D(3) receptor activity, and beta-adrenoceptor activity on the loss chasing of age and IQ-matched healthy adults randomized to treatment or an appropriate control/placebo. In Experiment 1, participants consumed amino-acid drinks that did or did not contain the serotonin precursor, tryptophan. In Experiment 2, participants received a single 176 μg dose of the D(2)/D(3) receptor agonist, pramipexole, or placebo. In Experiment 3, participants received a single 80 mg dose of the beta-adrenoceptor blocker, propranolol, or placebo. Following treatment, participants completed a computerized loss-chasing game. Mood and heart rate were measured at baseline and following treatment. Tryptophan depletion significantly reduced the number of decisions made to chase losses, and the number of consecutive decisions to chase, in the absence of marked changes in mood. By contrast, pramipexole significantly increased the value of losses chased and diminished the value of losses surrendered. Propranolol markedly reduced heart rate, but produced no significant changes in loss-chasing behavior. Loss chasing can be thought of as an aversively motivated escape behavior controlled, in part, by the marginal value of continued gambling relative to the value of already accumulated losses. Serotonin and dopamine appear to play dissociable roles in the tendency of individuals to gamble to recover, or to seek to 'escape' from, previous losses. Serotonergic activity seems to promote the availability of loss chasing as a behavioral option, whereas D(2)/D(3) receptor activity produces complex changes in the value of losses judged worth chasing. Sympathetic arousal, at least as mediated by beta-adrenoceptors, does not play a major role in laboratory-based loss-chasing choices.

Altered probabilistic learning and response biases in schizophrenia: behavioral evidence and neurocomputational modeling

OBJECTIVE

Patients with schizophrenia (SZ) show reinforcement learning impairments related to both the gradual/procedural acquisition of reward contingencies, and the ability to use trial-to-trial feedback to make rapid behavioral adjustments.

METHOD

We used neurocomputational modeling to develop plausible mechanistic hypotheses explaining reinforcement learning impairments in individuals with SZ. We tested the model with a novel Go/NoGo learning task in which subjects had to learn to respond or withhold responses when presented with different stimuli associated with different probabilities of gains or losses in points. We analyzed data from 34 patients and 23 matched controls, characterizing positive- and negative-feedback-driven learning in both a training phase and a test phase.

RESULTS

Consistent with simulations from a computational model of aberrant dopamine input to the basal ganglia patients, patients with SZ showed an overall increased rate of responding in the training phase, together with reduced response-time acceleration to frequently rewarded stimuli across training blocks, and a reduced relative preference for frequently rewarded training stimuli in the test phase. Patients did not differ from controls on measures of procedural negative-feedback-driven learning, although patients with SZ exhibited deficits in trial-to-trial adjustments to negative feedback, with these measures correlating with negative symptom severity.

CONCLUSIONS

These findings support the hypothesis that patients with SZ have a deficit in procedural "Go" learning, linked to abnormalities in DA transmission at D1-type receptors, despite a "Go bias" (increased response rate), potentially related to excessive tonic dopamine. Deficits in trial-to-trial reinforcement learning were limited to a subset of patients with SZ with severe negative symptoms, putatively stemming from prefrontal cortical dysfunction.

Neurogenetics and pharmacology of learning, motivation, and cognition

Many of the individual differences in cognition, motivation, and learning-and the disruption of these processes in neurological conditions-are influenced by genetic factors. We provide an integrative synthesis across human and animal studies, focusing on a recent spate of evidence implicating a role for genes controlling dopaminergic function in frontostriatal circuitry, including COMT, DARPP-32, DAT1, DRD2, and DRD4. These genetic effects are interpreted within theoretical frameworks developed in the context of the broader cognitive and computational neuroscience literature, constrained by data from pharmacological, neuroimaging, electrophysiological, and patient studies. In this framework, genes modulate the efficacy of particular neural computations, and effects of genetic variation are revealed by assays designed to be maximally sensitive to these computations. We discuss the merits and caveats of this approach and outline a number of novel candidate genes of interest for future study.

To head or to heed? Beyond the surface of selective action inhibition: a review

To head rather than heed to temptations is easier said than done. Since tempting actions are often contextually inappropriate, selective suppression is invoked to inhibit such actions. Thus far, laboratory tasks have not been very successful in highlighting these processes. We suggest that this is for three reasons. First, it is important to dissociate between an early susceptibility to making stimulus-driven impulsive but erroneous actions, and the subsequent selective suppression of these impulses that facilitates the selection of the correct action. Second, studies have focused on mean or median reaction times (RT), which conceals the temporal dynamics of action control. Third, studies have focused on group means, while considering individual differences as a source of error variance. Here, we present an overview of recent behavioral and imaging studies that overcame these limitations by analyzing RT distributions. As will become clear, this approach has revealed variations in inhibitory control over impulsive actions as a function of task instructions, conflict probability, and between-trial adjustments (following conflict or following an error trial) that are hidden if mean RTs are analyzed. Next, we discuss a selection of behavioral as well as imaging studies to illustrate that individual differences are meaningful and help understand selective suppression during action selection within samples of young and healthy individuals, but also within clinical samples of patients diagnosed with attention deficit/hyperactivity disorder or Parkinson's disease.

Supplementary motor area exerts proactive and reactive control of arm movements

Adaptive behavior requires the ability to flexibly control actions. This can occur either proactively to anticipate task requirements, or reactively in response to sudden changes. Here we report neuronal activity in the supplementary motor area (SMA) that is correlated with both forms of behavioral control. Single-unit and multiunit activity and intracranial local field potentials (LFPs) were recorded in macaque monkeys during a stop-signal task, which elicits both proactive and reactive behavioral control. The LFP power in high- (60-150 Hz) and low- (25-40 Hz) frequency bands was significantly correlated with arm movement reaction time, starting before target onset. Multiunit and single-unit activity also showed a significant regression with reaction time. In addition, LFPs and multiunit and single-unit activity changed their activity level depending on the trial history, mirroring adjustments on the behavioral level. Together, these findings indicate that neuronal activity in the SMA exerts proactive control of arm movements by adjusting the level of motor readiness. On trials when the monkeys successfully canceled arm movements in response to an unforeseen stop signal, the LFP power, particularly in a low (10-50 Hz) frequency range, increased early enough to be causally related to the inhibition of the arm movement on those trials. This indicated that neuronal activity in the SMA is also involved in response inhibition in reaction to sudden task changes. Our findings indicate, therefore, that SMA plays a role in the proactive control of motor readiness and the reactive inhibition of unwanted movements.

Conditions for the generation of beta oscillations in the subthalamic nucleus-globus pallidus network

The advance of Parkinson's disease is associated with the existence of abnormal oscillations within the basal ganglia with frequencies in the beta band (13-30 Hz). While the origin of these oscillations remains unknown, there is some evidence suggesting that oscillations observed in the basal ganglia arise due to interactions of two nuclei: the subthalamic nucleus (STN) and the globus pallidus pars externa (GPe). To investigate this hypothesis, we develop a computational model of the STN-GPe network based upon anatomical and electrophysiological studies. Significantly, our study shows that for certain parameter regimes, the model intrinsically oscillates in the beta range. Through an analytical study of the model, we identify a simple set of necessary conditions on model parameters that guarantees the existence of beta oscillations. These conditions for generation of oscillations are described by a set of simple inequalities and can be summarized as follows: (1) The excitatory connections from STN to GPe and the inhibitory connections from GPe to STN need to be sufficiently strong. (2) The time required by neurons to react to their inputs needs to be short relative to synaptic transmission delays. (3) The excitatory input from the cortex to STN needs to be high relative to the inhibition from striatum to GPe. We confirmed the validity of these conditions via numerical simulation. These conditions describe changes in parameters that are consistent with those expected as a result of the development of Parkinson's disease, and predict manipulations that could inhibit the pathological oscillations.

Neurally constrained modeling of perceptual decision making

Stochastic accumulator models account for response time in perceptual decision-making tasks by assuming that perceptual evidence accumulates to a threshold. The present investigation mapped the firing rate of frontal eye field (FEF) visual neurons onto perceptual evidence and the firing rate of FEF movement neurons onto evidence accumulation to test alternative models of how evidence is combined in the accumulation process. The models were evaluated on their ability to predict both response time distributions and movement neuron activity observed in monkeys performing a visual search task. Models that assume gating of perceptual evidence to the accumulating units provide the best account of both behavioral and neural data. These results identify discrete stages of processing with anatomically distinct neural populations and rule out several alternative architectures. The results also illustrate the use of neurophysiological data as a model selection tool and establish a novel framework to bridge computational and neural levels of explanation.

Subthalamic nucleus stimulation influences expression and suppression of impulsive behaviour in Parkinson's disease

Past studies show beneficial as well as detrimental effects of subthalamic nucleus deep-brain stimulation on impulsive behaviour. We address this paradox by investigating individuals with Parkinson's disease treated with subthalamic nucleus stimulation (n = 17) and healthy controls without Parkinson's disease (n = 17) on performance in a Simon task. In this reaction time task, conflict between premature response impulses and goal-directed action selection is manipulated. We applied distributional analytic methods to separate the strength of the initial response impulse from the proficiency of inhibitory control engaged subsequently to suppress the impulse. Patients with Parkinson's disease were tested when stimulation was either turned on or off. Mean conflict interference effects did not differ between controls and patients, or within patients when stimulation was on versus off. In contrast, distributional analyses revealed two dissociable effects of subthalamic nucleus stimulation. Fast response errors indicated that stimulation increased impulsive, premature responding in high conflict situations. Later in the reaction process, however, stimulation improved the proficiency with which inhibitory control was engaged to suppress these impulses selectively, thereby facilitating selection of the correct action. This temporal dissociation supports a conceptual framework for resolving past paradoxical findings and further highlights that dynamic aspects of impulse and inhibitory control underlying goal-directed behaviour rely in part on neural circuitry inclusive of the subthalamic nucleus.

The effect of Parkinson's disease on the dynamics of on-line and proactive cognitive control during action selection

Processing irrelevant visual information sometimes activates incorrect response impulses. The engagement of cognitive control mechanisms to suppress these impulses and make proactive adjustments to reduce the future impact of incorrect impulses may rely on the integrity of frontal-basal ganglia circuitry. Using a Simon task, we investigated the effects of basal ganglia dysfunction produced by Parkinson's disease (PD) on both on-line (within-trial) and proactive (between-trial) control efforts to reduce interference produced by the activation of an incorrect response. As a novel feature, we applied distributional analyses, guided by the activation-suppression model, to differentiate the strength of incorrect response activation and the proficiency of suppression engaged to counter this activation. For situations requiring on-line control, PD (n = 52) and healthy control (n = 30) groups showed similar mean interference effects (i.e., Simon effects) on reaction time (RT) and accuracy. Distributional analyses showed that although the strength of incorrect response impulses was similar between the groups PD patients were less proficient at suppressing these impulses. Both groups demonstrated equivalent and effective proactive control of response interference on mean RT and accuracy rates. However, PD patients were less effective at reducing the strength of incorrect response activation proactively. Among PD patients, motor symptom severity was associated with difficulties in on-line, but not in proactive, control of response impulses. These results suggest that basal ganglia dysfunction produced by PD has selective effects on cognitive control mechanisms engaged to resolve response conflict, with primary deficits in the on-line suppression of incorrect responses occurring in the context of a relatively spared ability to adjust control proactively to minimize future conflict.

Brain areas consistently linked to individual differences in perceptual decision-making in younger as well as older adults before and after training

Perceptual decision-making performance depends on several cognitive and neural processes. Here, we fit Ratcliff's diffusion model to accuracy data and reaction-time distributions from one numerical and one verbal two-choice perceptual-decision task to deconstruct these performance measures into the rate of evidence accumulation (i.e., drift rate), response criterion setting (i.e., boundary separation), and peripheral aspects of performance (i.e., nondecision time). These theoretical processes are then related to individual differences in brain activation by means of multiple regression. The sample consisted of 24 younger and 15 older adults performing the task in fMRI before and after 100 daily 1-hr behavioral training sessions in a multitude of cognitive tasks. Results showed that individual differences in boundary separation were related to striatal activity, whereas differences in drift rate were related to activity in the inferior parietal lobe. These associations were not significantly modified by adult age or perceptual expertise. We conclude that the striatum is involved in regulating response thresholds, whereas the inferior parietal lobe might represent decision-making evidence related to letters and numbers.

Cortical and subcortical interactions during action reprogramming and their related white matter pathways

The right inferior frontal gyrus (rIFG) and the presupplementary motor area (pre-SMA) have been identified with cognitive control-the top-down influence on other brain areas when nonroutine behavior is required. It has been argued that they "inhibit" habitual motor responses when environmental changes mean a different response should be made. However, whether such "inhibition" can be equated with inhibitory physiological interactions has been unclear, as has the areas' relationship with each other and the anatomical routes by which they influence movement execution. Paired-pulse transcranial magnetic stimulation (ppTMS) was applied over rIFG and primary motor cortex (M1) or over pre-SMA and M1 to measure their interactions, at a subsecond scale, during either inhibition and reprogramming of actions or during routine action selection. Distinct patterns of functional interaction between pre-SMA and M1 and between rIFG and M1 were found that were specific to action reprogramming trials; at a physiological level, direct influences of pre-SMA and rIFG on M1 were predominantly facilitatory and inhibitory, respectively. In a subsequent experiment, it was shown that the rIFG's inhibitory influence was dependent on pre-SMA. A third experiment showed that pre-SMA and rIFG influenced M1 at two time scales. By regressing white matter fractional anisotropy from diffusion-weighted magnetic resonance images against TMS-measured functional connectivity, it was shown that short-latency (6 ms) and longer latency (12 ms) influences were mediated by cortico-cortical and subcortical pathways, respectively, with the latter passing close to the subthalamic nucleus.

Overcoming status quo bias in the human brain

Humans often accept the status quo when faced with conflicting choice alternatives. However, it is unknown how neural pathways connecting cognition with action modulate this status quo acceptance. Here we developed a visual detection task in which subjects tended to favor the default when making difficult, but not easy, decisions. This bias was suboptimal in that more errors were made when the default was accepted. A selective increase in subthalamic nucleus (STN) activity was found when the status quo was rejected in the face of heightened decision difficulty. Analysis of effective connectivity showed that inferior frontal cortex, a region more active for difficult decisions, exerted an enhanced modulatory influence on the STN during switches away from the status quo. These data suggest that the neural circuits required to initiate controlled, nondefault actions are similar to those previously shown to mediate outright response suppression. We conclude that specific prefrontal-basal ganglia dynamics are involved in rejecting the default, a mechanism that may be important in a range of difficult choice scenarios.

Role of supplementary eye field in saccade initiation: executive, not direct, control

The goal of this study was to determine whether the activity of neurons in the supplementary eye field (SEF) is sufficient to control saccade initiation in macaque monkeys performing a saccade countermanding (stop signal) task. As previously observed, many neurons in the SEF increase the discharge rate before saccade initiation. However, when saccades are canceled in response to a stop signal, effectively no neurons with presaccadic activity display discharge rate modulation early enough to contribute to saccade cancellation. Moreover, SEF neurons do not exhibit a specific threshold discharge rate that could trigger saccade initiation. Yet, we observed more subtle relations between SEF activation and saccade production. The activity of numerous SEF neurons was correlated with response time and varied with sequential adjustments in response latency. Trials in which monkeys canceled or produced a saccade in a stop signal trial were distinguished by a modest difference in discharge rate of these SEF neurons before stop signal or target presentation. These findings indicate that neurons in the SEF, in contrast to counterparts in the frontal eye field and superior colliculus, do not contribute directly and immediately to the initiation of visually guided saccades. However the SEF may proactively regulate saccade production by biasing the balance between gaze-holding and gaze-shifting based on prior performance and anticipated task requirements.

Stimulation of the subthalamic nucleus and impulsivity: release your horses

OBJECTIVE

In Parkinson disease (PD) patients, deep brain stimulation (DBS) of the subthalamic nucleus (STN) may contribute to certain impulsive behavior during high-conflict decisions. A neurocomputational model of the basal ganglia has recently been proposed that suggests this behavioral aspect may be related to the role played by the STN in relaying a "hold your horses" signal intended to allow more time to settle on the best option. The aim of the present study was 2-fold: 1) to extend these observations by providing evidence that the STN may influence and prevent the execution of any response even during low-conflict decisions; and 2) to identify the neural correlates of this effect.

METHODS

We measured regional cerebral blood flow during a Go/NoGo and a control (Go) task to study the motor improvement and response inhibition deficits associated with STN-DBS in patients with PD.

RESULTS

Although it improved Unified Parkinson Disease Rating Scale motor ratings and induced a global decrease in reaction time during task performance, STN-DBS impaired response inhibition, as revealed by an increase in commission errors in NoGo trials. These behavioral effects were accompanied by changes in synaptic activity consisting of a reduced activation in the cortical networks responsible for reactive and proactive response inhibition.

INTERPRETATION

The present results suggest that although it improves motor functions in PD patients, modulation of STN hyperactivity with DBS may tend at the same time to favor the appearance of impulsive behavior by acting on the gating mechanism involved in response initiation.

Neurocomputational models of motor and cognitive deficits in Parkinson's disease

We review the contributions of biologically constrained computational models to our understanding of motor and cognitive deficits in Parkinson's disease (PD). The loss of dopaminergic neurons innervating the striatum in PD, and the well-established role of dopamine (DA) in reinforcement learning (RL), enable neural network models of the basal ganglia (BG) to derive concrete and testable predictions. We focus in this review on one simple underlying principle - the notion that reduced DA increases activity and causes long-term potentiation in the indirect pathway of the BG. We show how this theory can provide a unified account of diverse and seemingly unrelated phenomena in PD including progressive motor degeneration as well as cognitive deficits in RL, decision making and working memory. DA replacement therapy and deep brain stimulation can alleviate some aspects of these impairments, but can actually introduce negative effects such as motor dyskinesias and cognitive impulsivity. We discuss these treatment effects in terms of modulation of specific mechanisms within the computational framework. In addition, we review neurocomputational interpretations of increased impulsivity in the face of response conflict in patients with deep-brain-stimulation.

Optimal performance in a countermanding saccade task

Countermanding an action is a fundamental form of cognitive control. In a saccade-countermanding task, subjects are instructed that, if a stop signal appears shortly after a target, they are to maintain fixation rather than to make a saccade to the target. In recent years, recordings in the frontal eye fields and superior colliculus of behaving non-human primates have found correlates of such countermanding behavior in movement and fixation neurons. In this work, we extend a previous neural network model of countermanding to account for the high pre-target activity of fixation neurons. We propose that this activity reflects the functioning of control mechanisms responsible for optimizing performance. We demonstrate, using computer simulations and mathematical analysis, that pre-target fixation neuronal activity supports countermanding behavior that maximizes reward rate as a function of the stop signal delay, fraction of stop signal trials, intertrial interval, duration of timeout, and relative reward value. We propose experiments to test these predictions regarding optimal behavior.

Initiation and termination of integration in a decision process

Experimental data indicate that simple motor decisions in vertebrates are preceded by integration of evidence in certain cortical areas, and that the competition between them is resolved in the basal ganglia. While the occurrence of cortical integration is well established, it is not yet clear exactly how the integration occurs. Several models have been proposed, including the race model, the feed forward inhibition (FFI) model and the leaky competing accumulator (LCA) model. In this paper we establish qualitative and quantitative differences between the above mentioned models, with respect to how they are able to initiate the integration process without integrating noise prior to stimulus onset, as well as the models' ability to terminate the integration after a decision has been made, to ensure the possibility of subsequent decisions. Our results show that the LCA model has advantages over the race model and the FFI model in both respects, leading to shorter decision times and an effective termination process.

Explicit melioration by a neural diffusion model

When faced with choices between two sources of reward, animals can rapidly adjust their rates of responding to each so that overall reinforcement increases. Herrnstein's 'matching law' provides a simple description of the equilibrium state of this choice allocation process: animals reallocate behavior so that relative rates of responding equal, or match, the relative rates of reinforcement obtained for each response. Herrnstein and colleagues proposed 'melioration' as a dynamical process for achieving this equilibrium, but left details of its operation unspecified. Here we examine a way of filling in the details that links the decision making and operant conditioning literatures and extends choice proportion predictions into predictions about inter-response times. Our approach implements melioration in an adaptive version of the drift diffusion model (DDM), which is widely used in decision making research to account for response time distributions. When the drift parameter of the DDM is 0 and its threshold parameters are inversely proportional to reward rates, its choice proportions dynamically track a state of exact matching. A DDM with fixed thresholds and drift that is determined by differences in reward rates can produce similar, but not identical, results. We examine the choice probability and inter-response time predictions of these models, separately and in combination, and the possible implications for brain organization provided by neural network implementations of them. Results suggest that melioration and matching may derive from synapses that estimate reward rates by a process of leaky integration, and that link together the input and output stages of a two-stage stimulus-response mechanism.

Instructional control of reinforcement learning: a behavioral and neurocomputational investigation

Humans learn how to behave directly through environmental experience and indirectly through rules and instructions. Behavior analytic research has shown that instructions can control behavior, even when such behavior leads to sub-optimal outcomes (Hayes, S. (Ed.). 1989. Rule-governed behavior: cognition, contingencies, and instructional control. Plenum Press.). Here we examine the control of behavior through instructions in a reinforcement learning task known to depend on striatal dopaminergic function. Participants selected between probabilistically reinforced stimuli, and were (incorrectly) told that a specific stimulus had the highest (or lowest) reinforcement probability. Despite experience to the contrary, instructions drove choice behavior. We present neural network simulations that capture the interactions between instruction-driven and reinforcement-driven behavior via two potential neural circuits: one in which the striatum is inaccurately trained by instruction representations coming from prefrontal cortex/hippocampus (PFC/HC), and another in which the striatum learns the environmentally based reinforcement contingencies, but is "overridden" at decision output. Both models capture the core behavioral phenomena but, because they differ fundamentally on what is learned, make distinct predictions for subsequent behavioral and neuroimaging experiments. Finally, we attempt to distinguish between the proposed computational mechanisms governing instructed behavior by fitting a series of abstract "Q-learning" and Bayesian models to subject data. The best-fitting model supports one of the neural models, suggesting the existence of a "confirmation bias" in which the PFC/HC system trains the reinforcement system by amplifying outcomes that are consistent with instructions while diminishing inconsistent outcomes.