Modeling functions of striatal dopamine modulation in learning and planning

BGRL: Basal Ganglia inspired Reinforcement Learning based framework for deep brain stimulators

Deep Brain Stimulation (DBS) is an implantable medical device used for electrical stimulation to treat neurological disorders. Traditional DBS devices provide fixed frequency pulses, but personalized adjustment of stimulation parameters is crucial for optimal treatment. This paper introduces a Basal Ganglia inspired Reinforcement Learning (BGRL) approach, incorporating a closed-loop feedback mechanism to suppress neural synchrony during neurological fluctuations. The BGRL approach leverages the resemblance between the Basal Ganglia region of brain by incorporating the actor-critic architecture of reinforcement learning (RL). Simulation results demonstrate that BGRL significantly reduces synchronous electrical pulses compared to other standard RL algorithms. BGRL algorithm outperforms existing RL methods in terms of suppression capability and energy consumption, validated through comparisons using ensemble oscillators. Results shown in the paper demonstrate BGRL suppressed the synchronous electrical pulses across three signaling regimes namely regular, chaotic and bursting by 40%, 146% and 40% respectively as compared to soft actor-critic model. BGRL shows promise in effectively suppressing neural synchrony in DBS therapy, providing an efficient alternative to open-loop methodologies.

Basal ganglia network dynamics and function: Role of direct, indirect and hyper-direct pathways in action selection

Basal ganglia (BG) are a widely recognized neural basis for action selection, but its decision-making mechanism is still a difficult problem for researchers. Therefore, we constructed a spiking neural network inspired by the BG anatomical data. Simulation experiments were based on the principle of dis-inhibition and our functional hypothesis within the BG: the direct pathway, the indirect pathway, and the hyper-direct pathway of the BG jointly implement the initiation execution and termination of motor programs. Firstly, we studied the dynamic process of action selection with the network, which contained intra-group competition and inter-group competition. Secondly, we focused on the effects of the stimulus intensity and the proportion of excitation and inhibition on the GPi/SNr. The results suggested that inhibition and excitation shape action selection. They also explained why the firing rate of GPi/SNr did not continue to increase in the action-selection experiment. Finally, we discussed the experimental results with the functional hypothesis. Uniquely, this paper summarized the decision-making neural mechanism of action selection based on the direct pathway, the indirect pathway, and the hyper-direct pathway within BG.

How Working Memory and Reinforcement Learning Are Intertwined: A Cognitive, Neural, and Computational Perspective

Reinforcement learning and working memory are two core processes of human cognition and are often considered cognitively, neuroscientifically, and algorithmically distinct. Here, we show that the brain networks that support them actually overlap significantly and that they are less distinct cognitive processes than often assumed. We review literature demonstrating the benefits of considering each process to explain properties of the other and highlight recent work investigating their more complex interactions. We discuss how future research in both computational and cognitive sciences can benefit from one another, suggesting that a key missing piece for artificial agents to learn to behave with more human-like efficiency is taking working memory's role in learning seriously. This review highlights the risks of neglecting the interplay between different processes when studying human behavior (in particular when considering individual differences). We emphasize the importance of investigating these dynamics to build a comprehensive understanding of human cognition.

Dopamine, Prediction Error and Beyond

A large body of work has linked dopaminergic signaling to learning and reward processing. It stresses the role of dopamine in reward prediction error signaling, a key neural signal that allows us to learn from past experiences, and that facilitates optimal choice behavior. Latterly, it has become clear that dopamine does not merely code prediction error size but also signals the difference between the expected value of rewards, and the value of rewards actually received, which is obtained through the integration of reward attributes such as the type, amount, probability and delay. More recent work has posited a role of dopamine in learning beyond rewards. These theories suggest that dopamine codes absolute or unsigned prediction errors, playing a key role in how the brain models associative regularities within its environment, while incorporating critical information about the reliability of those regularities. Work is emerging supporting this perspective and, it has inspired theoretical models of how certain forms of mental pathology may emerge in relation to dopamine function. Such pathology is frequently related to disturbed inferences leading to altered internal models of the environment. Thus, it is critical to understand the role of dopamine in error-related learning and inference.

Noradrenergic and dopaminergic involvement in novelty modulation of aversive memory generalization of adult rats

The generalization of aversive memory can be defined as the phenomenon in which a situation similar to (but distinctive from) a previous aversive event triggers an avoidance response. This phenomenon has been suggested to play a role in several psychological disorders. In this study, we investigate the effects of novelty on the generalization of fear memories, and the involvement of noradrenergic and dopaminergic systems in this process. For this study we used male Wistar rats (3 months old, 300-400 g). The participation of each neurotransmitter system was evaluated separately (two set of experiments). In each experimental set, the animals were divided in groups (8 animals each): (i) control, (ii) novelty, and, (iii) antagonist + novelty group (timolol, a β-adrenergic antagonist, or SCH23390, a D1/D5 dopaminergic antagonist, in the first and in the second set of experiments, respectively). Additionaly, to investigate whether novelty exposure increases the levels of noradrenaline and/or dopamine in the hippocampus fifteen animals were divided in three groups (5 animals each).: (i) naïve, (ii) control; and, (iii) novelty. To examine aversive memory, and generalization of aversive memory, we trained adult male Wistar rats in an inhibitory avoidance (IA) memory task and after in a modified inhibitory avoidance (MIA). Before the MIA training some of the animals were exposed to environmental novelty (open field). Immediately before this novelty exposure, some animals received intrahippocampal infusion of timolol (β-adrenergic antagonist), SCH23390 (D1/D5 antagonist) or vehicle to evaluate the involvement of noradrenergic and dopaminergic systems. Finally, to evaluate aversive memory and generalization of aversive memory respectively, half of the animals in each group were tested on IA and half on MIA. We confirmed that the exposure to novelty blocks the generalization of aversive memory, but moreover, demonstrated that this process involves activation of β-adrenergic and dopaminergic D1/D5 pathways. We additionally observed that exposure to novelty raises hippocampal levels of noradrenaline and dopamine. This suggests that these neurotransmitters not only influence long-term memory (LTM) as such, but also aversive memory generalization.

Beyond drug-induced alteration of glutamate homeostasis, astrocytes may contribute to dopamine-dependent intrastriatal functional shifts that underlie the development of drug addiction: A working hypothesis

The transition from recreational drug use to compulsive drug‐seeking habits, the hallmark of addiction, has been shown to depend on a shift in the locus of control over behaviour from the ventral to the dorsolateral striatum. This process has hitherto been considered to depend on the aberrant engagement of dopamine‐dependent plasticity processes within neuronal networks. However, exposure to drugs of abuse also triggers cellular and molecular adaptations in astrocytes within the striatum which could potentially contribute to the intrastriatal transitions observed during the development of drug addiction. Pharmacological interventions aiming to restore the astrocytic mechanisms responsible for maintaining homeostatic glutamate concentrations in the nucleus accumbens, that are altered by chronic exposure to addictive drugs, abolish the propensity to relapse in both preclinical and, to a lesser extent, clinical studies. Exposure to drugs of abuse also alters the function of astrocytes in the dorsolateral striatum, wherein dopaminergic mechanisms control drug‐seeking habits, associated compulsivity and relapse. This suggests that drug‐induced alterations in the glutamatergic homeostasis maintained by astrocytes throughout the entire striatum may interact with dopaminergic mechanisms to promote aberrant plasticity processes that contribute to the maintenance of maladaptive drug‐seeking habits. Capitalising on growing evidence that astrocytes play a fundamental regulatory role in glutamate and dopamine transmission in the striatum, we present an innovative model of a quadripartite synaptic microenvironment within which astrocytes channel functional interactions between the dopaminergic and glutamatergic systems that may represent the primary striatal functional unit that undergoes drug‐induced adaptations eventually leading to addiction.

Born to learn: The inspiration, progress, and future of evolved plastic artificial neural networks

Biological neural networks are systems of extraordinary computational capabilities shaped by evolution, development, and lifelong learning. The interplay of these elements leads to the emergence of biological intelligence. Inspired by such intricate natural phenomena, Evolved Plastic Artificial Neural Networks (EPANNs) employ simulated evolution in-silico to breed plastic neural networks with the aim to autonomously design and create learning systems. EPANN experiments evolve networks that include both innate properties and the ability to change and learn in response to experiences in different environments and problem domains. EPANNs' aims include autonomously creating learning systems, bootstrapping learning from scratch, recovering performance in unseen conditions, testing the computational advantages of particular neural components, and deriving hypotheses on the emergence of biological learning. Thus, EPANNs may include a large variety of different neuron types and dynamics, network architectures, plasticity rules, and other factors. While EPANNs have seen considerable progress over the last two decades, current scientific and technological advances in artificial neural networks are setting the conditions for radically new approaches and results. Exploiting the increased availability of computational resources and of simulation environments, the often challenging task of hand-designing learning neural networks could be replaced by more autonomous and creative processes. This paper brings together a variety of inspiring ideas that define the field of EPANNs. The main methods and results are reviewed. Finally, new opportunities and possible developments are presented.

A system-level mathematical model of Basal Ganglia motor-circuit for kinematic planning of arm movements

In this paper, a novel system-level mathematical model of the Basal Ganglia (BG) for kinematic planning, is proposed. An arm composed of several segments presents a geometric redundancy. Thus, selecting one trajectory among an infinite number of possible ones requires overcoming redundancy, according to some kinds of optimization. Solving this optimization is assumed to be the function of BG in planning. In the proposed model, first, a mathematical solution of kinematic planning is proposed for movements of a redundant arm in a plane, based on minimizing energy consumption. Next, the function of each part in the model is interpreted as a possible role of a nucleus of BG. Since the kinematic variables are considered as vectors, the proposed model is presented based on the vector calculus. This vector model predicts different neuronal populations in BG which is in accordance with some recent experimental studies. According to the proposed model, the function of the direct pathway is to calculate the necessary rotation of each joint, and the function of the indirect pathway is to control each joint rotation considering the movement of the other joints. In the proposed model, the local feedback loop between Subthalamic Nucleus and Globus Pallidus externus is interpreted as a local memory to store the previous amounts of movements of the other joints, which are utilized by the indirect pathway. In this model, activities of dopaminergic neurons would encode, at short-term, the error between the desired and actual positions of the end-effector. The short-term modulating effect of dopamine on Striatum is also modeled as cross product. The model is simulated to generate the commands of a redundant manipulator. The performance of the model is studied for different reaching movements between 8 points in a plane. Finally, some symptoms of Parkinson's disease such as bradykinesia and akinesia are simulated by modifying the model parameters, inspired by the dopamine depletion.

Molecular Mechanism and Dynamics of S-Deoxyephedrine Moving through Molecular Channels within D3R

In this article, the trajectories of S-deoxyephedrine (SBD) along molecular channels within the complex protein structure of third dopamine receptor (D₃R) are analyzed via molecular dynamic techniques, including potential mean force calculations of umbrella samplings from the 4.5 version of the GROMACS program. Changes in free energy due to the movement of SBD within D₃R are determined, and the molecular dynamic mechanisms of SBD transmitting along molecular channels are probed. Molecular simulated results show that the change in free energy is calculated as 171.7 kJ·mol^-1 for the transmission of SBD toward the outside of the cell along the y+ axis functional molecular channel and is 275.0 kJ·mol^-1 for movement toward the intracellular structure along the y- axis. Within the internal structure of D₃R, the changes in free energy are determined to be 103.6, 242.1, 459.7, and 127.8 kJ·mol^-1 for transmission of SBD along the x+, x-, z+, and z- axes, respectively, toward the cell bilayer membrane, which indicates that SBD leaves much more easily along the x+ axis through the gap between the TM5 (the fifth transmembrane helix) and TM6 (the sixth transmembrane helix) from the internal structure of D₃R. The values of free-energy changes indicate that SBD molecules can clear the protective channel within D₃R, which helps dopamine molecules to leave the D₃R internal structure along the x+ axis and to prevent them for exerting excessive neurotransmitter function. Therefore, our results suggest that SBD is effective for development as a drug for treating schizophrenia and its pharmacology is closely related to its dynamics and mechanisms within the molecular pathway of dopamine receptors.

Molecular dynamics simulation of d-Benzedrine transmitting through molecular channels within D3R

Dex-Benzedrine (known as d-Benzedrine or SAT) acts in dopamine receptors of central nerve cell system. In clinic, SAT is used to treat a variety of diseases; meanwhile, it has dependence and addiction. In order to investigate the pharmacology and addiction mechanisms of SAT as a medicine, in this paper, we have studied the structure of D₃R complex protein with SAT, and based on which, using potential mean force with umbrella samplings and the simulated phospholipid bilayer membrane (or POPC bilayer membrane), the molecular dynamics simulation was performed to obtain free energy changes upon the trajectories for SAT moving along the molecular channels within D₃R. The free energy change for SAT transmitting toward the outside of cell along the functional molecular channel within D₃R is 83.5 kJ mol^-1. The change of free energy for SAT to permeate into the POPC bilayer membrane along the protective molecular channel within D₃R is 87.7 kJ mol^-1. Our previous work gave that the free energy for Levo-Benzedrine (RAT) transmitting toward the outside of cell along the functional molecular channel within D₃R is 91.4 kJ mol^-1, while it is 117.7 kJ mol^-1 for RAT to permeate into the POPC bilayer membrane along the protective molecular channel within D₃R. The values of free energy suggest that SAT relatively prefers likely to pass through the functional molecular channel within D₃R for increasing the release of dopamine molecules resulting in a variety of functional effects for SAT. The obtained results show that the pharmacology and addiction mechanisms of SAT as a drug are closely related to the molecular dynamics and mechanism for SAT transmitting along molecular channels within D₃R.

Neurotransmitters and Novelty: A Systematic Review

Our brains are highly responsive to novelty. However, how novelty is processed in the brain, and what neurotransmitter systems play a role therein, remains elusive. Here, we systematically review studies on human participants that have looked at the neuromodulatory basis of novelty detection and processing. While theoretical models and studies on nonhuman animals have pointed to a role of the dopaminergic, cholinergic, noradrenergic and serotonergic systems, the human literature has focused almost exclusively on the first two. Dopamine was found to affect electrophysiological responses to novelty early in time after stimulus presentation, but evidence on its effects on later processing was found to be contradictory: While neuropharmacological studies mostly yielded null effects, gene studies did point to an important role for dopamine. Acetylcholine seems to dampen novelty signals in the medial temporal lobe, but boost them in frontal cortex. Findings on 5-HT (serotonin) were found to be mostly contradictory. Two large gaps were identified in the literature. First, few studies have looked at neuromodulatory influences on behavioral effects of novelty. Second, no study has looked at the involvement of the noradrenergic system in novelty processing.

Intrinsic plasticity: an emerging player in addiction

Exposure to drugs of abuse, such as cocaine, leads to plastic changes in the activity of brain circuits, and a prevailing view is that these changes play a part in drug addiction. Notably, there has been intense focus on drug-induced changes in synaptic excitability and much less attention on intrinsic excitability factors (that is, excitability factors that are remote from the synapse). Accumulating evidence now suggests that intrinsic factors such as K+ channels are not only altered by cocaine but may also contribute to the shaping of the addiction phenotype.

Computational models of basal-ganglia pathway functions: focus on functional neuroanatomy

Over the past 15 years, computational models have had a considerable impact on basal-ganglia research. Most of these models implement multiple distinct basal-ganglia pathways and assume them to fulfill different functions. As there is now a multitude of different models, it has become complex to keep track of their various, sometimes just marginally different assumptions on pathway functions. Moreover, it has become a challenge to oversee to what extent individual assumptions are corroborated or challenged by empirical data. Focusing on computational, but also considering non-computational models, we review influential concepts of pathway functions and show to what extent they are compatible with or contradict each other. Moreover, we outline how empirical evidence favors or challenges specific model assumptions and propose experiments that allow testing assumptions against each other.

Akt1 deficiency modulates reward learning and reward prediction error in mice

In contemporary reinforcement learning models, reward prediction error (RPE), the difference between the expected and actual reward, is thought to guide action value learning through the firing activity of dopaminergic neurons. Given the importance of dopamine in reward learning and the involvement of Akt1 in dopamine-dependent behaviors, the aim of this study was to investigate whether Akt1 deficiency modulates reward learning and the magnitude of RPE using Akt1 mutant mice as a model. In comparison to wild-type littermate controls, the expression of Akt1 proteins in mouse brains occurred in a gene-dosage-dependent manner and Akt1 heterozygous (HET) mice exhibited impaired striatal Akt1 activity under methamphetamine challenge. No genotypic difference was found in the basal levels of dopamine and its metabolites. In a series of reward-related learning tasks, HET mice displayed a relatively efficient method of updating reward information from the environment during the acquisition phase of the two natural reward tasks and in the reverse section of the dynamic foraging T-maze but not in methamphetamine-induced or aversive-related reward learning. The implementation of a standard reinforcement learning model and the Bayesian hierarchical parameter estimation show that HET mice have higher RPE magnitudes and that their action values are updated more rapidly among all three test sections in T-maze. These results indicate that Akt1 deficiency modulates natural reward learning and RPE. This study showed a promising avenue for investigating RPE in mutant mice and provided evidence for the potential link from genetic deficiency, to neurobiological abnormalities, to impairment in higher-order cognitive functioning.

Aggression as positive reinforcement in people with intellectual disabilities

From an applied behavior-analytic perspective, aggression in people with intellectual disabilities is mostly maintained by social reinforcement consequences. However, nonsocial consequences have also been identified in functional assessments on aggression. Behaviors producing their own reinforcement have been labeled "automatic" or "nonsocial" in the behavior-analytic literature, a label that bares a striking resemblance to biobehavioral explanations of reward-seeking behaviors. Biobehavioral studies have revealed that aggression activates the same endogenous brain mechanisms as primary reinforcers like food. Therefore, integrating brain-environment explanations would result in a better understanding of the functional mechanisms associated with nonsocial aggression. The purpose of this paper was to explore aggression as a reinforcing consequence for reinforcement-seeking behaviors in people with intellectual disabilities. First, the literature establishing aggression as reinforcement for arbitrary responding will be reviewed. Next, the reward-related biological process associated with aggression was described. Finally, the paper discusses what might be done to assess and treat aggression maintained by nonsocial reinforcement.

Dissociation of response and feedback negativity in schizophrenia: electrophysiological and computational evidence for a deficit in the representation of value

Contrasting theories of schizophrenia propose that the disorder is characterized by a deficit in phasic changes in dopamine activity in response to ongoing events or, alternatively, by a weakness in the representation of the value of responses. Schizophrenia patients have reliably reduced brain activity following incorrect responses but other research suggests that they may have intact feedback-related potentials, indicating that the impairment may be specifically response-related. We used event-related brain potentials and computational modeling to examine this issue by comparing the neural response to outcomes with the neural response to behaviors that predict outcomes in patients with schizophrenia and psychiatrically healthy comparison subjects. We recorded feedback-related activity in a passive gambling task and a time estimation task and error-related activity in a flanker task. Patients' brain activity following an erroneous response was reduced compared to comparison subjects but feedback-related activity did not differ between groups. To test hypotheses about the possible causes of this pattern of results, we used computational modeling of the electrophysiological data to simulate the effects of an overall reduction in patients' sensitivity to feedback, selective insensitivity to positive or negative feedback, reduced learning rate, and a decreased representation of the value of the response given the stimulus on each trial. The results of the computational modeling suggest that schizophrenia patients exhibit weakened representation of response values, possibly due to failure of the basal ganglia to strongly associate stimuli with appropriate response alternatives.

An imperfect dopaminergic error signal can drive temporal-difference learning

An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.

What are the physiological changes that take place in the brain when we solve a problem or learn a new skill? It is commonly assumed that behavior adaptations are realized on the microscopic level by changes in synaptic efficacies. However, this is hard to verify experimentally due to the difficulties of identifying the relevant synapses and monitoring them over long periods during a behavioral task. To address this question computationally, we develop a spiking neuronal network model of actor-critic temporal-difference learning, a variant of reinforcement learning for which neural correlates have already been partially established. The network learns a complex task by means of an internally generated reward signal constrained by recent findings on the dopaminergic system. Our model combines top-down and bottom-up modelling approaches to bridge the gap between synaptic plasticity and system-level learning. It paves the way for further investigations of the dopaminergic system in reward learning in the healthy brain and in pathological conditions such as Parkinson's disease, and can be used as a module in functional models based on brain-scale circuitry.

Computational cognitive models of prefrontal-striatal-hippocampal interactions in Parkinson's disease and schizophrenia

Disruption to different components of the prefrontal cortex, basal ganglia, and hippocampal circuits leads to various psychiatric and neurological disorders including Parkinson's disease (PD) and schizophrenia. Medications used to treat these disorders (such as levodopa, dopamine agonists, antipsychotics, among others) affect the prefrontal-striatal-hippocampal circuits in a complex fashion. We have built models of prefrontal-striatal and striatal-hippocampal interactions which simulate cognitive dysfunction in PD and schizophrenia. In these models, we argue that the basal ganglia is key for stimulus-response learning, the hippocampus for stimulus-stimulus representational learning, and the prefrontal cortex for stimulus selection during learning about multidimensional stimuli. In our models, PD is associated with reduced dopamine levels in the basal ganglia and prefrontal cortex. In contrast, the cognitive deficits in schizophrenia are associated primarily with hippocampal dysfunction, while the occurrence of negative symptoms is associated with frontostriatal deficits in a subset of patients. In this paper, we review our past models and provide new simulation results for both PD and schizophrenia. We also describe an extended model that includes simulation of the different functional role of D1 and D2 dopamine receptors in the basal ganglia and prefrontal cortex, a dissociation we argue is essential for understanding the non-uniform effects of levodopa, dopamine agonists, and antipsychotics on cognition. Motivated by clinical and physiological data, we discuss model limitations and challenges to be addressed in future models of these brain disorders.

The small-conductance calcium-activated potassium channel is a key modulator of firing and long-term depression in the dorsal striatum

The striatum is considered to be critical for the control of goal-directed action, with the lateral dorsal striatum (latDS) being implicated in modulation of habits and the nucleus accumbens thought to represent a limbic-motor interface. Although medium spiny neurons from different striatal subregions exhibit many similar properties, differential firing and synaptic plasticity could contribute to the varied behavioral roles across subregions. Here, we examined the contribution of small-conductance calcium-activated potassium channels (SKs) to action potential generation and synaptic plasticity in adult rat latDS and nucleus accumbens shell (NAS) projection neurons in vitro. The SK-selective antagonist apamin exerted a prominent effect on latDS firing, significantly decreasing the interspike interval. Furthermore, prolonged latDS depolarization increased the interspike interval and reduced firing, and this enhancement was reversed by apamin. In contrast, NAS neurons exhibited greater basal firing rates and less regulation of firing by SK inhibition and prolonged depolarization. LatDS neurons also had greater SK currents than NAS neurons under voltage-clamp. Importantly, SK inhibition with apamin facilitated long-term depression (LTD) induction in the latDS but not the NAS, without alterations in glutamate release. In addition, SK activation in the latDS prevented LTD induction. Greater SK function in the latDS than in the NAS was not secondary to differences in sodium or inwardly rectifying potassium channel function, and apamin enhancement of firing did not reflect indirect action through cholinergic interneurons. Thus, these data demonstrate that SKs are potent modulators of action potential generation and LTD in the dorsal striatum, and could represent a fundamental cellular mechanism through which habits are regulated.

Conditional routing of information to the cortex: a model of the basal ganglia's role in cognitive coordination

The basal ganglia play a central role in cognition and are involved in such general functions as action selection and reinforcement learning. Here, we present a model exploring the hypothesis that the basal ganglia implement a conditional information-routing system. The system directs the transmission of cortical signals between pairs of regions by manipulating separately the selection of sources and destinations of information transfers. We suggest that such a mechanism provides an account for several cognitive functions of the basal ganglia. The model also incorporates a possible mechanism by which subsequent transfers of information control the release of dopamine. This signal is used to produce novel stimulus-response associations by internalizing transferred cortical representations in the striatum. We discuss how the model is related to production systems and cognitive architectures. A series of simulations is presented to illustrate how the model can perform simple stimulus-response tasks, develop automatic behaviors, and provide an account of impairments in Parkinson's and Huntington's diseases.

Computational models of reinforcement learning: the role of dopamine as a reward signal

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.

Aging and the neuroeconomics of decision making: A review

Neuroeconomics refers to a combination of paradigms derived from neuroscience, psychology, and economics for the study of decision making and is an area that has received considerable scientific attention in the recent literature. Using realistic laboratory tasks, researchers seek to study the neurocognitive processes underlying economic decision making and outcome-based decision learning, as well as individual differences in these processes and the social and affective factors that modulate them. To this point, one question has remained largely unanswered: What happens to decision-making processes and their neural substrates during aging? After all, aging is associated with neurocognitive change, which may affect outcome-based decision making. In our study, we use the subjective expected utility model-a well-established decision-making model in economics-as a descriptive framework. After a short survey of the brain areas and neurotransmitter systems associated with outcome-based decision making-and of the effects of aging thereon-we review a number of decision-making studies. Their general data pattern indicates that the decision-making process is changed by age: The elderly perform less efficiently than younger participants, as demonstrated, for instance, by the smaller total rewards that the elderly acquire in lab tasks. These findings are accounted for in terms of age-related deficiencies in the probability and value parameters of the subjective expected utility model. Finally, we discuss some implications and suggestions for future research.

Short-term gains, long-term pains: how cues about state aid learning in dynamic environments

Successful investors seeking returns, animals foraging for food, and pilots controlling aircraft all must take into account how their current decisions will impact their future standing. One challenge facing decision makers is that options that appear attractive in the short-term may not turn out best in the long run. In this paper, we explore human learning in a dynamic decision making task which places short- and long-term rewards in conflict. Our goal in these studies was to evaluate how people's mental representation of a task affects their ability to discover an optimal decision strategy. We find that perceptual cues that readily align with the underlying state of the task environment help people overcome the impulsive appeal of short-term rewards. Our experimental manipulations, predictions, and analyses are motivated by current work in reinforcement learning which details how learners value delayed outcomes in sequential tasks and the importance that "state" identification plays in effective learning.

Inhibitory contribution to suprathreshold corticostriatal responses: an experimental and modeling study

Neostriatal neurons may undergo events of spontaneous synchronization as those observed in recurrent networks of excitatory neurons, even when cortical afferents are transected. It is necessary to explain these events because the neostriatum is a recurrent network of inhibitory neurons. Synchronization of neuronal activity may be caused by plateau-like depolarizations. Plateau-like orthodromic depolarizations that resemble up-states in medium spiny neostriatal neurons (MSNs) may be induced by a single corticostriatal suprathreshold stimulus. Slow synaptic depolarizations may last hundreds of milliseconds, decay slower than the monosynaptic glutamatergic synaptic potentials that induce them, and sustain repetitive firing. Because inhibitory inputs impinging onto MSNs have a reversal potential above the resting membrane potential but below the threshold for firing, they conform a type of "shunting inhibition". This work asks if shunting GABAergic inputs onto MSNs arrive asynchronously enough as to help in sustaining the plateau-like corticostriatal response after a single cortical stimulus. This may help to begin explaining autonomous processing in the striatal micro-circuitry in the presence of a tonic excitatory drive and independently of spatio-temporally organized inputs. It is shown here that besides synaptic currents from AMPA/KA- and NMDA-receptors, as well as L-type intrinsic Ca(2+)- currents, inhibitory synapses help in maintaining the slow depolarization, although they accomplish the role of depressing firing at the beginning of the response. We then used a NEURON model of spiny cells to show that inhibitory synapses arriving asynchronously on the dendrites can help to simulate a plateau potential similar to that observed experimentally.

Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective

Research on human and animal behavior has long emphasized its hierarchical structure-the divisibility of ongoing behavior into discrete tasks, which are comprised of subtask sequences, which in turn are built of simple actions. The hierarchical structure of behavior has also been of enduring interest within neuroscience, where it has been widely considered to reflect prefrontal cortical functions. In this paper, we reexamine behavioral hierarchy and its neural substrates from the point of view of recent developments in computational reinforcement learning. Specifically, we consider a set of approaches known collectively as hierarchical reinforcement learning, which extend the reinforcement learning paradigm by allowing the learning agent to aggregate actions into reusable subroutines or skills. A close look at the components of hierarchical reinforcement learning suggests how they might map onto neural structures, in particular regions within the dorsolateral and orbital prefrontal cortex. It also suggests specific ways in which hierarchical reinforcement learning might provide a complement to existing psychological models of hierarchically structured behavior. A particularly important question that hierarchical reinforcement learning brings to the fore is that of how learning identifies new action routines that are likely to provide useful building blocks in solving a wide range of future problems. Here and at many other points, hierarchical reinforcement learning offers an appealing framework for investigating the computational and neural underpinnings of hierarchically structured behavior.

A dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism

Parkinson's disease (PD) patients exhibit cognitive deficits, including reinforcement learning, working memory (WM) and set shifting. Computational models of the basal ganglia-frontal system posit similar mechanisms for these deficits in terms of reduced dynamic range of striatal dopamine (DA) signals in both medicated and non-medicated states. Here, we report results from the first study that tests PD patients on and off dopaminergic medications in a modified version of the AX continuous performance (AX-CPT) working memory task, consisting of three performance phases and one phase requiring WM associations to be learned via reinforcement feedback. Patients generally showed impairments relative to controls. Medicated patients showed deficits specifically when having to ignore distracting stimuli during the delay. Our models suggest that this impairment is due to medication causing excessive WM updating by enhancing striatal "Go" signals that facilitate such updating, while concurrently suppressing "NoGo" signals. In contrast, patients off medication showed deficits consistent with an overall reduction in striatal DA and associated Go updating signals. Supporting this dichotomy, patients on and off medication both showed attentional shifting deficits, but for different reasons. Deficits in non-medicated patients were consistent with an inability to update the new attentional set, whereas those in medicated patients were evident when having to ignore distractors that had previously been task relevant. Finally, in the feedback-based WM phase, medicated patients were better than unmedicated patients, suggesting a key role of striatal DA in using feedback to update information into WM. These results lend further insight into the role of basal ganglia dopamine in WM and broadly support predictions from neurocomputational models.

Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model

A critical problem in daily decision making is how to choose actions now in order to bring about rewards later. Indeed, many of our actions have long-term consequences, and it is important to not be myopic in balancing the pros and cons of different options, but rather to take into account both immediate and delayed consequences of actions. Failures to do so may be manifest as persistent, maladaptive decision-making, one example of which is addiction where behavior seems to be driven by the immediate positive experiences with drugs, despite the delayed adverse consequences. A recent study by Takahashi et al. (2007) investigated the effects of cocaine sensitization on decision making in rats and showed that drug use resulted in altered representations in the ventral striatum and the dorsolateral striatum, areas that have been implicated in the neural instantiation of a computational solution to optimal long-term actions selection called the Actor/Critic framework. In this Focus article we discuss their results and offer a computational interpretation in terms of drug-induced impairments in the Critic. We first survey the different lines of evidence linking the subparts of the striatum to the Actor/Critic framework, and then suggest two possible scenarios of breakdown that are suggested by Takahashi et al.'s (2007) data. As both are compatible with the current data, we discuss their different predictions and how these could be empirically tested in order to further elucidate (and hopefully inch towards curing) the neural basis of drug addiction.

A model of reward choice based on the theory of reinforcement learning

A model explaining behavioral "impulsivity" and "self-control" is proposed on the basis of the theory of reinforcement learning. The discount coefficient gamma, which in this theory accounts for the subjective reduction in the value of a delayed reinforcement, is identified with the overall level of dopaminergic neuron activity which, according to published data, also determines the behavioral variant. Computer modeling showed that high values of gamma are characteristic of predominantly "self-controlled" subjects, while smaller values of gamma are characteristic of "impulsive" subjects.

Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transferred to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the delta-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications.

Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system

The prefrontal cortex (PFC) has long been thought to serve as an 'executive' that controls the selection of actions and cognitive functions more generally. However, the mechanistic basis of this executive function has not been clearly specified often amounting to a homunculus. This paper reviews recent attempts to deconstruct this homunculus by elucidating the precise computational and neural mechanisms underlying the executive functions of the PFC. The overall approach builds upon existing mechanistic models of the basal ganglia (BG) and frontal systems known to play a critical role in motor control and action selection, where the BG provide a 'Go' versus 'NoGo' modulation of frontal action representations. In our model, the BG modulate working memory representations in prefrontal areas to support more abstract executive functions. We have developed a computational model of this system that is capable of developing human-like performance on working memory and executive control tasks through trial-and-error learning. This learning is based on reinforcement learning mechanisms associated with the midbrain dopaminergic system and its activation via the BG and amygdala. Finally, we briefly describe various empirical tests of this framework.

A role of the basal ganglia and midbrain nuclei for initiation of motor sequences

The mesial premotor cortex is crucial for planning sequential procedures and movement initiation. With event-related (ER) functional magnetic resonance imaging (fMRI) it has been possible to separate mesial premotor activation before, during, and after self-initiated movements and, thereby, to distinguish advance planning from execution. The mesial premotor cortex is part of distributed cortico-basal ganglia-thalamo-cortical networks but, to date, the subcortical contributions to self-initiated movements are far less well understood. Using ER fMRI at 3T in 12 right-handed male volunteers, we studied the subcortical activation preceding an automated four-digit finger sequence that was either self-initiated or triggered externally by a visual cue. Beyond typical cortical activation increases in fronto-parietal regions, both initiation modes induced consistent subcortical activation in basal ganglia, midbrain (substantia nigra), and ipsilateral cerebellum. The planning phase of the internally initiated condition, when contrasted with the externally triggered condition, was associated with enhanced activity in frontal regions (mesial premotor cortex/rostral cingulate zone, dorsolateral prefrontal cortex), parietal regions (precuneus, inferior parietal cortex, encroaching onto V5/MT), insula, contralateral anterior putamen and midbrain (bilateral red nucleus/subthalamic nucleus). These data demonstrate the impact of initiation mode on planning-related activity in the ventral basal ganglia and interconnected midbrain nuclei, thereby stressing the crucial role of distributed cortico-basal ganglia-thalamo-cortical networks for self-initiated automated motor repertoires. Involvement of the substantia nigra during planning, as shown here, indicates dopaminergic gating of motor sequences.

A neural network model of hippocampal-striatal-prefrontal interactions in contextual conditioning

The hippocampus is thought to be critical for encoding contextually bound memories and setting the context for ongoing behavior. However, the mechanisms by which the hippocampal-cortical system controls behavior are poorly understood. We propose a computational model in which the hippocampus exerts contextual control over motivated behavior by gating prefrontal cortex inputs to the nucleus accumbens. The model integrates the episodic memory functions of the hippocampus, the prefrontal role in representing the motivational stimuli and cognitive control, and the role of striatal regions in conditioned learning within a single theoretical framework. Simulation results are consistent with the hypothesis that hippocampal-prefrontal interactions may act as the neural substrate that allows contextual cues to override conditioned responses at the level of the nucleus accumbens. Prefrontal and hippocampal input overrides the predominant CS-US association if the context is inconsistent, and promotes flexible selection of previously learned associations and behaviors. Simulated transection of the fornix, effectively eliminating hippocampal and prefrontal influence over the nucleus accumbens, abolishes normal contextual modulation of behavior. The model is consistent with a wide range of empirical data.

Turning off cortical ensembles stops striatal Up states and elicits phase perturbations in cortical and striatal slow oscillations in rat in vivo

In vivo, cortical neurons and striatal medium spiny neurons (MSN) display robust subthreshold depolarizations (Up states) during which they are enabled to fire action potentials. In the cortex, Up states are believed to occur simultaneously in a neuronal ensemble and to be sustained by local network interactions. It is known that MSN are impelled into the Up state by extra-striatal (primarily cortical) inputs, but the mechanisms that sustain and determine the end of striatal Up states are still debated. Furthermore, it has not been established if brisk perturbations of ongoing cortical oscillations alter rhythmic transitions between Up and Down states in striatal neurons. Here we report that MSN Up states terminate abruptly when persistent activity in cortical ensembles providing afferents to a given striatal region is turned off by local electrical stimulation or ends spontaneously. In addition, we found that phase perturbations in MSN membrane potential slow oscillations induced by cortical stimulation replicate the stimulus-induced dynamics of spiking activity in cortical ensembles. Overall, these results suggest that striatal Up states are single-cell subthreshold representations of episodes of persistent spiking in cortical ensembles. A precise spatial and temporal alignment between episodes of cortical persistent activity and striatal Up states would allow MSN to detect specific cortical inputs embedded within a more general cortical signal.

Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia

The prefrontal cortex has long been thought to subserve both working memory (the holding of information online for processing) and executive functions (deciding how to manipulate working memory and perform processing). Although many computational models of working memory have been developed, the mechanistic basis of executive function remains elusive, often amounting to a homunculus. This article presents an attempt to deconstruct this homunculus through powerful learning mechanisms that allow a computational model of the prefrontal cortex to control both itself and other brain areas in a strategic, task-appropriate manner. These learning mechanisms are based on subcortical structures in the midbrain, basal ganglia, and amygdala, which together form an actor-critic architecture. The critic system learns which prefrontal representations are task relevant and trains the actor, which in turn provides a dynamic gating mechanism for controlling working memory updating. Computationally, the learning mechanism is designed to simultaneously solve the temporal and structural credit assignment problems. The model's performance compares favorably with standard backpropagation-based temporal learning mechanisms on the challenging 1-2-AX working memory task and other benchmark working memory tasks.

Conditioned locomotion induced by unilateral intrastriatal administration of apomorphine: D(2) receptor activation is critical but not the expression of the unconditioned response

The present study examined the role of D(1) and D(2) receptors in the conditioning of apomorphine-induced locomotor behavior. A Pavlovian conditioning protocol was used in which rats received 5 daily intrastriatal apomorphine treatments paired or unpaired to an open-field environment followed, 2 days later, by a saline test for conditioning. In the conditioning induction phase, the intrastriatal apomorphine treatment increased locomotor activity expressed as an increased number of sectional crossings and rearings. In the conditioning test, the apomorphine-paired group had significantly higher locomotor activity than the unpaired and vehicle groups, consistent with the development of a conditioned locomotor response. The concomitant blockade of D(1) and D(2) receptors with D(1) (SCH23390) and D(2) (sulpiride) antagonists prevented the apomorphine-induced behavioral response during the induction phase and in the conditioning test. Pretreatment with the D(1) antagonist SCH 23390 also blocked the apomorphine-induced behavioral response during the induction phase but did not block the apomorphine conditioned response. Pretreatment with the selective D(2) antagonist sulpiride blocked the apomorphine behavioral response during the induction phase and in the conditioning test. Altogether, these results indicate that antagonism of either the D(1) or D(2) receptors in the dorsal striatum can block apomorphine-induced locomotor activation but that D(2) but not D(1) antagonism can prevent the development of the apomorphine conditioned response. Altogether, these findings indicate a key role for the D(2) receptor site in the mediation of apomorphine conditioned behavior; and, in addition, that apomorphine conditioned locomotor response can develop without the expression of the locomotor stimulant response during the induction phase of conditioning.

Dopamine, prediction error and associative learning: a model-based account

The notion of prediction error has established itself at the heart of formal models of animal learning and current hypotheses of dopamine function. Several interpretations of prediction error have been offered, including the model-free reinforcement learning method known as temporal difference learning (TD), and the important Rescorla-Wagner (RW) learning rule. Here, we present a model-based adaptation of these ideas that provides a good account of empirical data pertaining to dopamine neuron firing patterns and associative learning paradigms such as latent inhibition, Kamin blocking and overshadowing. Our departure from model-free reinforcement learning also offers: 1) a parsimonious distinction between tonic and phasic dopamine functions; 2) a potential generalization of the role of phasic dopamine from valence-dependent "reward" processing to valence-independent "salience" processing; 3) an explanation for the selectivity of certain dopamine manipulations on motivation for distal rewards; and 4) a plausible link between formal notions of prediction error and accounts of disturbances of thought in schizophrenia (in which dopamine dysfunction is strongly implicated). The model distinguishes itself from existing accounts by offering novel predictions pertaining to the firing of dopamine neurons in various untested behavioral scenarios.

Involvement of the transmitter systems of the neostriatum in automation of motor skills in dogs

Chronic experiments were performed on six dogs to address the influence of automation of a motor skill on the components of an operant defensive reflex, rearrangement of posture, and the characteristics of the projection of the center of mass on a tensometric platform by the forepaws; the effects of activation and blockade of muscarinic and dopamine receptors on the process of automation of the motor skill were also studied. Despite high criteria for execution of the operant reflex, the reproducibility of reflex performance, increases in the tonic component of the response, and the clear diagonal pattern of the postural rearrangement, a local (as opposed to diffuse) projection of the position of the center of mass on the tensometric platform by the forepaws was seen only after prolonged training leading to automation of the skill. The effect of automation of the operant response could be obtained immediately following bilateral microinjection of the muscarinic receptor agonist carbachol into the neostriatum. The same effect, but weaker, was obtained using bilateral microinjections of the dopamine D2 receptor blocker raclopride into the neostriatum. Conversely, bilateral microinjections of the selective muscarinic M1 receptor blocker pirenzepine into the neostriatum gave the opposite result, with increases in the phasic component of the response, impairment of the diagonal pattern of the postural rearrangement, and diffuseness of the projection of the center of mass on the tensometric platform by the forelimbs. It is concluded that the indirect efferent output of the neostriatum has an important role in the process of learning the motor operant reflex involving maintenance of a defined degree of flexion (where the main role is played by the tonic component of the movement) and during the process of automation of the motor skill.

Banishing the homunculus: making working memory work

The prefrontal cortex has long been thought to subserve both working memory and "executive" function, but the mechanistic basis of their integrated function has remained poorly understood, often amounting to a homunculus. This paper reviews the progress in our laboratory and others pursuing a long-term research agenda to deconstruct this homunculus by elucidating the precise computational and neural mechanisms underlying these phenomena. We outline six key functional demands underlying working memory, and then describe the current state of our computational model of the prefrontal cortex and associated systems in the basal ganglia (BG). The model, called PBWM (prefrontal cortex, basal ganglia working memory model), relies on actively maintained representations in the prefrontal cortex, which are dynamically updated/gated by the basal ganglia. It is capable of developing human-like performance largely on its own by taking advantage of powerful reinforcement learning mechanisms, based on the midbrain dopaminergic system and its activation via the basal ganglia and amygdala. These learning mechanisms enable the model to learn to control both itself and other brain areas in a strategic, task-appropriate manner. The model can learn challenging working memory tasks, and has been corroborated by several important empirical studies.

Comparison of population activity in the dorsal premotor cortex and putamen during the learning of arbitrary visuomotor mappings

A previous study found that as monkeys learned novel mappings between visual cues and responses, neuronal activity patterns evolved at approximately the same time in both the dorsal premotor cortex (PMd) and the putamen. Here we report that, in both regions, the population activity for novel mappings came to resemble that for familiar ones as learning progressed. Both regions showed activity differences on trials with correct responses versus those with incorrect ones. In addition to these common features, we observed two noteworthy differences between PMd and putamen activity during learning. After a response choice had been made, but prior to feedback about the correctness of that choice (reward or nonreward), the putamen showed a sustained activity increase in activity, whereas PMd did not. Also in the putamen, this prereward activity was highly selective for the specific visuomotor mapping that had just been performed, and this selectivity was maintained until the time of the reward. After performance reached an asymptote, the degree of this selectivity decreased markedly to the level typical for familiar visuomotor mappings. These findings support the hypothesis that neurons in the striatum play a pivotal role in associative learning.

Computational modeling and exploration of contour integration for visual saliency

We propose a computational model of contour integration for visual saliency. The model uses biologically plausible devices to simulate how the representations of elements aligned collinearly along a contour in an image are enhanced. Our model adds such devices as a dopamine-like fast plasticity, local GABAergic inhibition and multi-scale processing of images. The fast plasticity addresses the problem of how neurons in visual cortex seem to be able to influence neurons they are not directly connected to, for instance, as observed in contour closure effect. Local GABAergic inhibition is used to control gain in the system without using global mechanisms which may be non-plausible given the limited reach of axonal arbors in visual cortex. The model is then used to explore not only its validity in real and artificial images, but to discover some of the mechanisms involved in processing of complex visual features such as junctions and end-stops as well as contours. We present evidence for the validity of our model in several phases, starting with local enhancement of only a few collinear elements. We then test our model on more complex contour integration images with a large number of Gabor elements. Sections of the model are also extracted and used to discover how the model might relate contour integration neurons to neurons that process end-stops and junctions. Finally, we present results from real world images. Results from the model suggest that it is a good current approximation of contour integration in human vision. As well, it suggests that contour integration mechanisms may be strongly related to mechanisms for detecting end-stops and junction points. Additionally, a contour integration mechanism may be involved in finding features for objects such as faces. This suggests that visual cortex may be more information efficient and that neural regions may have multiple roles.

A Computational Model of the Functional Role of the Ventral-Striatal D2 Receptor in the Expression of Previously Acquired Behaviors

The functional role of dopamine has attracted a great deal of interest ever since it was empirically discovered that dopamine-blocking drugs could be used to treat psychosis. Specifically, the D2 receptor and its expression in the ventral striatum have emerged as pivotal in our understanding of the complex role of the neuromodulator in schizophrenia, reward, and motivation. Our departure from the ubiquitous temporal difference (TD) model of dopamine neuron firing allows us to account for a range of experimental evidence suggesting that ventral striatal dopamine D2 receptor manipulation selectively modulates motivated behavior for distal versus proximal outcomes. Whether an internal model or the TD approach (or a mixture) is better suited to a comprehensive exposition of tonic and phasic dopamine will have important implications for our understanding of reward, motivation, schizophrenia, and impulsivity. We also use the model to help unite some of the leading cognitive hypotheses of dopamine function under a computational umbrella. We have used the model ourselves to stimulate and focus new rounds of experimental research.

A model of antipsychotic action in conditioned avoidance: a computational approach

The selective ability of antipsychotic drugs (APDs) to attenuate conditioned avoidance responding (CAR) has been recognized for over 50 years. However, most efforts to account for this finding have been either neurochemically oriented (focusing on the neuromodulator dopamine) or behavioral, with little effort invested in uniting the two within a computational model. In this paper we propose a computational model, based on concepts from formal reinforcement learning theory, which accounts for the basic finding that noncataleptic doses of APDs disrupt avoidance without disrupting escape. The model formally separates out sensory, motor, and reward processes, and makes novel predictions pertaining to the dose- and time-dependent effects of APDs on response latencies--predictions which we verified in experimental studies using four different APDs (haloperidol, chlorpromazine, risperidone, and clozapine). The APD action in this model is most consistent with an effect on 'expected future reward'--an idea closely linked to motivational drives and consistent with several leading theories of dopamine action.