Functional properties of the basal ganglia's re-entrant loop architecture: selection and reinforcement

Roles of nucleus accumbens CREB and dynorphin in dysregulation of motivation

Psychostimulants such as amphetamine and cocaine are believed to produce dependence by causing rapid, supraphysiological elevations in synaptic dopamine (DA) within the nucleus accumbens (NAc) (Volkow et al. 2009, Neuropharmacology 56: 3-8). These changes in forebrain DA transmission are similar to those evoked by natural reinforcers (Louilot et al. 1991, Brain Res 553: 313-317; Roitman et al. 2004, J Neurosci 24: 1265-1271), but are of greater magnitude and longer duration. Repeated drug exposure causes compensatory neuroadaptations in neurons of the NAc, some of which may modulate excess DA in a homeostatic fashion. One such adaptation is the activation of the transcription factor CREB (cAMP response element-binding protein) within neurons of the NAc. Although elevated levels of transcriptionally active CREB appear to attenuate DA transmission by increasing expression of the endogenous κ opioid receptor (KOR) ligand dynorphin, increased dynorphin transmission may ultimately have undesirable effects that contribute to drug withdrawal states as well as comorbid psychiatric illnesses such as depression. This state may prompt a return to drug use to mitigate the adverse effects of withdrawal. This article summarizes our current understanding of how CREB and dynorphin contribute to the dysregulation of motivation and describes novel therapeutic strategies that derive from preclinical research in this area.

Synchrony of anterior cingulate cortex and insular-striatal activation predicts ambiguity aversion in individuals with low impulsivity

Personal attitude toward ambiguity contributes to individual differences in decision making in uncertain situations. Operationally, these attitudes reflect the various coping strategies elected to overcome the limited information. A key brain region involved in cognitive control for performance adjustments is the dorsal anterior cingulate cortex (dACC). To test how dACC functional network connectivity would be modulated by uncertainty and differ between individuals, 24 healthy participants underwent functional MRI in 3 sequential runs: 1 resting-state and 2 decision-making task runs. Individuals with lower nonplanning impulsiveness made greater use of a Pass option and avoided uncertain ambiguous situations. Seed-based functional connectivity analysis during the task runs revealed that stronger activation synchrony between the left dACC and the right anterior insula correlated with greater use of a Pass response option. During the resting-state, stronger resting-state functional connectivity between the left dACC and the ventral striatum predicted the adoption of Pass as a behavioral strategy and correlated with stronger task-activated synchrony between the dACC and the right anterior insula. Our findings indicate that that the synchrony between the dACC and insula-striatal circuitry was greater in individuals with low compared with high nonplanning impulsiveness and contributed to adopting Pass as a useful behavioral strategy.

Phasic dopamine as a prediction error of intrinsic and extrinsic reinforcements driving both action acquisition and reward maximization: a simulated robotic study

An important issue of recent neuroscientific research is to understand the functional role of the phasic release of dopamine in the striatum, and in particular its relation to reinforcement learning. The literature is split between two alternative hypotheses: one considers phasic dopamine as a reward prediction error similar to the computational TD-error, whose function is to guide an animal to maximize future rewards; the other holds that phasic dopamine is a sensory prediction error signal that lets the animal discover and acquire novel actions. In this paper we propose an original hypothesis that integrates these two contrasting positions: according to our view phasic dopamine represents a TD-like reinforcement prediction error learning signal determined by both unexpected changes in the environment (temporary, intrinsic reinforcements) and biological rewards (permanent, extrinsic reinforcements). Accordingly, dopamine plays the functional role of driving both the discovery and acquisition of novel actions and the maximization of future rewards. To validate our hypothesis we perform a series of experiments with a simulated robotic system that has to learn different skills in order to get rewards. We compare different versions of the system in which we vary the composition of the learning signal. The results show that only the system reinforced by both extrinsic and intrinsic reinforcements is able to reach high performance in sufficiently complex conditions.

How prediction errors shape perception, attention, and motivation

Prediction errors (PE) are a central notion in theoretical models of reinforcement learning, perceptual inference, decision-making and cognition, and prediction error signals have been reported across a wide range of brain regions and experimental paradigms. Here, we will make an attempt to see the forest for the trees and consider the commonalities and differences of reported PE signals in light of recent suggestions that the computation of PE forms a fundamental mode of brain function. We discuss where different types of PE are encoded, how they are generated, and the different functional roles they fulfill. We suggest that while encoding of PE is a common computation across brain regions, the content and function of these error signals can be very different and are determined by the afferent and efferent connections within the neural circuitry in which they arise.

Functional neuroanatomy of the basal ganglia

The "basal ganglia" refers to a group of subcortical nuclei responsible primarily for motor control, as well as other roles such as motor learning, executive functions and behaviors, and emotions. Proposed more than two decades ago, the classical basal ganglia model shows how information flows through the basal ganglia back to the cortex through two pathways with opposing effects for the proper execution of movement. Although much of the model has remained, the model has been modified and amplified with the emergence of new data. Furthermore, parallel circuits subserve the other functions of the basal ganglia engaging associative and limbic territories. Disruption of the basal ganglia network forms the basis for several movement disorders. This article provides a comprehensive account of basal ganglia functional anatomy and chemistry and the major pathophysiological changes underlying disorders of movement. We try to answer three key questions related to the basal ganglia, as follows: What are the basal ganglia? What are they made of? How do they work? Some insight on the canonical basal ganglia model is provided, together with a selection of paradoxes and some views over the horizon in the field.

Intrinsically motivated action-outcome learning and goal-based action recall: a system-level bio-constrained computational model

Reinforcement (trial-and-error) learning in animals is driven by a multitude of processes. Most animals have evolved several sophisticated systems of 'extrinsic motivations' (EMs) that guide them to acquire behaviours allowing them to maintain their bodies, defend against threat, and reproduce. Animals have also evolved various systems of 'intrinsic motivations' (IMs) that allow them to acquire actions in the absence of extrinsic rewards. These actions are used later to pursue such rewards when they become available. Intrinsic motivations have been studied in Psychology for many decades and their biological substrates are now being elucidated by neuroscientists. In the last two decades, investigators in computational modelling, robotics and machine learning have proposed various mechanisms that capture certain aspects of IMs. However, we still lack models of IMs that attempt to integrate all key aspects of intrinsically motivated learning and behaviour while taking into account the relevant neurobiological constraints. This paper proposes a bio-constrained system-level model that contributes a major step towards this integration. The model focusses on three processes related to IMs and on the neural mechanisms underlying them: (a) the acquisition of action-outcome associations (internal models of the agent-environment interaction) driven by phasic dopamine signals caused by sudden, unexpected changes in the environment; (b) the transient focussing of visual gaze and actions on salient portions of the environment; (c) the subsequent recall of actions to pursue extrinsic rewards based on goal-directed reactivation of the representations of their outcomes. The tests of the model, including a series of selective lesions, show how the focussing processes lead to a faster learning of action-outcome associations, and how these associations can be recruited for accomplishing goal-directed behaviours. The model, together with the background knowledge reviewed in the paper, represents a framework that can be used to guide the design and interpretation of empirical experiments on IMs, and to computationally validate and further develop theories on them.

Establishing causality for dopamine in neural function and behavior with optogenetics

Dopamine (DA) is known to play essential roles in neural function and behavior. Accordingly, DA neurons have been the focus of intense experimental investigation that has led to many important advances in our understanding of how DA influences these processes. However, it is becoming increasingly appreciated that delineating the precise contributions of DA neurons to cellular, circuit, and systems-level phenomena will require more sophisticated control over their patterns of activity than conventional techniques can provide. Specifically, the roles played by DA neurons are likely to depend on their afferent and efferent connectivity, the timing and length of their neural activation, and the nature of the behavior under investigation. Recently developed optogenetic tools hold great promise for disentangling these complex issues. Here we discuss the use of light-sensitive microbial opsins in the context of outstanding questions in DA research. A major technical advance offered by these proteins is the ability to bidirectionally modulate DA neuron activity in in vitro and in vivo preparations on a time scale that more closely approximates those of neural, perceptual and behavioral events. In addition, continued advances in rodent genetics and viral-mediated gene delivery have contributed to the ability to selectively target DA neurons or their individual afferent and efferent connections. Further, these tools are suitable for use in experimental subjects engaged in complex behaviors. After reviewing the strengths and limitations of optogenetic methodologies, we conclude by describing early efforts in the application of this valuable new approach that demonstrate its potential to improve our understanding of the neural and behavioral functions of DA. This article is part of a Special Issue entitled Optogenetics (7th BRES).

Thorndike's Law 2.0: Dopamine and the Regulation of Thrift

Dopamine is widely associated with reward, motivation, and reinforcement learning. Research on dopamine has emphasized its contribution to compulsive behaviors, such as addiction and overeating, with less examination of its potential role in behavioral flexibility in normal, non-pathological states. In the study reviewed here, we investigated the effect of increased tonic dopamine in a two-lever homecage operant paradigm where the relative value of the levers was dynamic, requiring the mice to constantly monitor reward outcome and adapt their behavior. The data were fit to a temporal difference learning model that showed that mice with elevated dopamine exhibited less coupling between reward history and behavioral choice. This work suggests a way to integrate motivational and learning theories of dopamine into a single formal model where tonic dopamine regulates the expression of prior reward learning by controlling the degree to which learned reward values bias behavioral choice. Here I place these results in a broader context of dopamine's role in instrumental learning and suggest a novel hypothesis that tonic dopamine regulates thrift, the degree to which an animal needs to exploit its prior reward learning to maximize return on energy expenditure. Our data suggest that increased dopamine decreases thriftiness, facilitating energy expenditure, and permitting greater exploration. Conversely, this implies that decreased dopamine increases thriftiness, favoring the exploitation of prior reward learning, and diminishing exploration. This perspective provides a different window onto the role dopamine may play in behavioral flexibility and its failure, compulsive behavior.

Ethanol seeking by Long Evans rats is not always a goal-directed behavior

Background

Two parallel and interacting processes are said to underlie animal behavior, whereby learning and performance of a behavior is at first via conscious and deliberate (goal-directed) processes, but after initial acquisition, the behavior can become automatic and stimulus-elicited (habitual). With respect to instrumental behaviors, animal learning studies suggest that the duration of training and the action-outcome contingency are two factors involved in the emergence of habitual seeking of “natural” reinforcers (e.g., sweet solutions, food or sucrose pellets). To rigorously test whether behaviors reinforced by abused substances such as ethanol, in particular, similarly become habitual was the primary aim of this study.

Methodology/Principal Findings

Male Long Evans rats underwent extended or limited operant lever press training with 10% sucrose/10% ethanol (10S10E) reinforcement (variable interval (VI) or (VR) ratio schedule of reinforcement), or with 10% sucrose (10S) reinforcement (VI schedule only). Once training and pretesting were complete, the impact of outcome devaluation on operant behavior was evaluated after lithium chloride injections were paired with the reinforcer, or unpaired 24 hours later. After limited, but not extended instrumental training, lever pressing by groups trained under VR with 10S10E and under VI with 10S was sensitive to outcome devaluation. In contrast, responding by both the extended and limited training 10S10E VI groups was not sensitive to ethanol devaluation during the test for habitual behavior.

Conclusions/Significance

Operant behavior by rats trained to self-administer an ethanol-sucrose solution showed variable sensitivity to a change in the value of ethanol, with relative insensitivity developing sooner in animals that received time-variable ethanol reinforcement during training sessions. One important implication, with respect to substance abuse in humans, is that initial learning about the relationship between instrumental actions and the opportunity to consume ethanol-containing drinks can influence the time course for the development or expression of habitual ethanol seeking behavior.

Building Bridges between Perceptual and Economic Decision-Making: Neural and Computational Mechanisms

Investigation into the neural and computational bases of decision-making has proceeded in two parallel but distinct streams. Perceptual decision-making (PDM) is concerned with how observers detect, discriminate, and categorize noisy sensory information. Economic decision-making (EDM) explores how options are selected on the basis of their reinforcement history. Traditionally, the sub-fields of PDM and EDM have employed different paradigms, proposed different mechanistic models, explored different brain regions, disagreed about whether decisions approach optimality. Nevertheless, we argue that there is a common framework for understanding decisions made in both tasks, under which an agent has to combine sensory information (what is the stimulus) with value information (what is it worth). We review computational models of the decision process typically used in PDM, based around the idea that decisions involve a serial integration of evidence, and assess their applicability to decisions between good and gambles. Subsequently, we consider the contribution of three key brain regions – the parietal cortex, the basal ganglia, and the orbitofrontal cortex (OFC) – to perceptual and EDM, with a focus on the mechanisms by which sensory and reward information are integrated during choice. We find that although the parietal cortex is often implicated in the integration of sensory evidence, there is evidence for its role in encoding the expected value of a decision. Similarly, although much research has emphasized the role of the striatum and OFC in value-guided choices, they may play an important role in categorization of perceptual information. In conclusion, we consider how findings from the two fields might be brought together, in order to move toward a general framework for understanding decision-making in humans and other primates.

Dopamine release in the basal ganglia

Dopamine (DA) is a key transmitter in the basal ganglia, yet DA transmission does not conform to several aspects of the classic synaptic doctrine. Axonal DA release occurs through vesicular exocytosis and is action potential- and Ca²⁺-dependent. However, in addition to axonal release, DA neurons in midbrain exhibit somatodendritic release by an incompletely understood, but apparently exocytotic, mechanism. Even in striatum, axonal release sites are controversial, with evidence for DA varicosities that lack postsynaptic specialization, and largely extrasynaptic DA receptors and transporters. Moreover, DA release is often assumed to reflect a global response to a population of activities in midbrain DA neurons, whether tonic or phasic, with precise timing and specificity of action governed by other basal ganglia circuits. This view has been reinforced by anatomical evidence showing dense axonal DA arbors throughout striatum, and a lattice network formed by DA axons and glutamatergic input from cortex and thalamus. Nonetheless, localized DA transients are seen in vivo using voltammetric methods with high spatial and temporal resolution. Mechanistic studies using similar methods in vitro have revealed local regulation of DA release by other transmitters and modulators, as well as by proteins known to be disrupted in Parkinson's disease and other movement disorders. Notably, the actions of most other striatal transmitters on DA release also do not conform to the synaptic doctrine, with the absence of direct synaptic contacts for glutamate, GABA, and acetylcholine (ACh) on striatal DA axons. Overall, the findings reviewed here indicate that DA signaling in the basal ganglia is sculpted by cooperation between the timing and pattern of DA input and those of local regulatory factors.