Neural mechanisms of acquired phasic dopamine responses in learning

Human Substantia Nigra Neurons Encode Reward Expectations

Dopamine (DA) signals originating from substantia nigra (SN) neurons are centrally involved in the regulation of motor and reward processing. DA signals behaviorally relevant events where reward outcomes differ from expectations (reward prediction errors, RPEs). RPEs play a crucial role in learning optimal courses of action and in determining response vigor when an agent expects rewards. Nevertheless, how reward expectations, crucial for RPE calculations, are conveyed to and represented in the dopaminergic system is not fully understood, especially in the human brain where the activity of DA neurons is difficult to study. One possibility, suggested by evidence from animal models, is that DA neurons explicitly encode reward expectations. Alternatively, they may receive RPE information directly from upstream brain regions. To address whether SN neuron activity directly reflects reward expectation information, we directly examined the encoding of reward expectation signals in human putative DA neurons by performing single-unit recordings from the SN of patients undergoing neurosurgery. Patients played a two-armed bandit decision-making task in which they attempted to maximize reward. We show that neuronal firing rates (FR) of putative DA neurons during the reward expectation period explicitly encode reward expectations. First, activity in these neurons was modulated by previous trial outcomes, such that FR were greater after positive outcomes than after neutral or negative outcome trials. Second, this increase in FR was associated with shorter reaction times, consistent with an invigorating effect of DA neuron activity during expectation. These results suggest that human DA neurons explicitly encode reward expectations, providing a neurophysiological substrate for a signal critical for reward learning.

Choice-selective sequences dominate in cortical relative to thalamic inputs to NAc to support reinforcement learning

How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about whether and how representations differ across these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of reinforcement-learning algorithms, in both a circuit model based on synaptic plasticity and one based on neural dynamics. Finally, we test and confirm a prediction of our circuit models by direct manipulation of nucleus accumbens input neurons.

The role of withdrawal in mesocorticolimbic drug cue reactivity in opioid use disorder

Opioid use disorder (OUD) is characterized by heightened cognitive, physiological, and neural responses to opioid-related cues that are mediated by mesocorticolimbic brain pathways. Craving and withdrawal are key symptoms of addiction that persist during physiological abstinence. The present study evaluated the relationship between the brain response to drug cues in OUD and baseline levels of craving and withdrawal. We used functional magnetic resonance imaging (fMRI) to examine brain responses to opioid-related pictures and control pictures in 29 OUD patients. Baseline measures of drug use severity, opioid craving, and withdrawal symptoms were assessed prior to cue exposure and correlated with subsequent brain responses to drug cues. Mediation analysis was conducted to test the indirect effect of drug use severity on brain cue reactivity through craving and withdrawal symptoms. We found that baseline drug use severity and opioid withdrawal symptoms, but not craving, were positively associated with the neural response to drug cues in the nucleus accumbens, orbitofrontal cortex, and amygdala. Withdrawal, but not craving, mediated the effect of drug use severity on the nucleus accumbens' response to drug cues. We did not find similar effects for the neural responses to stimuli unrelated to drugs. Our findings emphasize the central role of withdrawal symptoms as the mediator between the clinical severity of OUD and the brain correlates of sensitization to opioid-related cues. They suggest that in OUD, baseline withdrawal symptoms signal a high vulnerability to drug cues.

Wave-like dopamine dynamics as a mechanism for spatiotemporal credit assignment

Significant evidence supports the view that dopamine shapes learning by encoding reward prediction errors. However, it is unknown whether striatal targets receive tailored dopamine dynamics based on regional functional specialization. Here, we report wave-like spatiotemporal activity patterns in dopamine axons and release across the dorsal striatum. These waves switch between activational motifs and organize dopamine transients into localized clusters within functionally related striatal subregions. Notably, wave trajectories were tailored to task demands, propagating from dorsomedial to dorsolateral striatum when rewards are contingent on animal behavior and in the opponent direction when rewards are independent of behavioral responses. We propose a computational architecture in which striatal dopamine waves are sculpted by inference about agency and provide a mechanism to direct credit assignment to specialized striatal subregions. Supporting model predictions, dorsomedial dopamine activity during reward-pursuit signaled the extent of instrumental control and interacted with reward waves to predict future behavioral adjustments.

Modulation of Dopamine for Adaptive Learning: A Neurocomputational Model

There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine mediated tasks that transcend the experimental phenomena it was initially designed to address.

Neural Mechanisms of Human Decision-Making

We present a theory and neural network model of the neural mechanisms underlying human decision-making. We propose a detailed model of the interaction between brain regions, under a proposer-predictor-actor-critic​ ​framework. This theory is based on detailed animal data and theories of action-selection. Those theories are adapted to serial operation to bridge levels of analysis and explain human decision-making. Task-relevant areas of cortex propose a candidate plan using fast, model-free, parallel neural computations. Other areas of cortex and medial temporal lobe can then predict​ likely outcomes of that plan in this situation. This optional prediction- (or model-) based computation can produce better accuracy and generalization, at the expense of speed. Next, linked regions of basal ganglia act​ to accept or reject the proposed plan based on its reward history in similar contexts. If that plan is rejected, the process repeats to consider a new option. The reward-prediction system acts as a critic​ to determine the value of the outcome relative to expectations and produce dopamine as a training signal for cortex and basal ganglia. By operating sequentially and hierarchically, the same mechanisms previously proposed for animal action-selection could explain the most complex human plans and decisions. We discuss explanations of model-based decisions, habitization, and risky behavior based on the computational model.

A systems-neuroscience model of phasic dopamine

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Unraveling the Mysteries of Motivation

Motivation plays a central role in human behavior and cognition but is not well captured by widely used artificial intelligence (AI) and computational modeling frameworks. This Opinion article addresses two central questions regarding the nature of motivation: what are the nature and dynamics of the internal goals that drive our motivational system and how can this system be sufficiently flexible to support our ability to rapidly adapt to novel situations, tasks, etc.? In reviewing existing systems and neuroscience research and theorizing on these questions, a wealth of insights to constrain the development of computational models of motivation can be found.

The Unexplored Territory of Neural Models: Potential Guides for Exploring the Function of Metabotropic Neuromodulation

The space of possible neural models is enormous and under-explored. Single cell computational neuroscience models account for a range of dynamical properties of membrane potential, but typically do not address network function. In contrast, most models focused on network function address the dimensions of excitatory weight matrices and firing thresholds without addressing the complexities of metabotropic receptor effects on intrinsic properties. There are many under-explored dimensions of neural parameter space, and the field needs a framework for representing what has been explored and what has not. Possible frameworks include maps of parameter spaces, or efforts to categorize the fundamental elements and molecules of neural circuit function. Here we review dimensions that are under-explored in network models that include the metabotropic modulation of synaptic plasticity and presynaptic inhibition, spike frequency adaptation due to calcium-dependent potassium currents, and afterdepolarization due to calcium-sensitive non-specific cation currents and hyperpolarization activated cation currents. Neuroscience research should more effectively explore possible functional models incorporating under-explored dimensions of neural function.

Synchronicity: The Role of Midbrain Dopamine in Whole-Brain Coordination

Midbrain dopamine seems to play an outsized role in motivated behavior and learning. Widely associated with mediating reward-related behavior, decision making, and learning, dopamine continues to generate controversies in the field. While many studies and theories focus on what dopamine cells encode, the question of how the midbrain derives the information it encodes is poorly understood and comparatively less addressed. Recent anatomical studies suggest greater diversity and complexity of afferent inputs than previously appreciated, requiring rethinking of prior models. Here, we elaborate a hypothesis that construes midbrain dopamine as implementing a Bayesian selector in which individual dopamine cells sample afferent activity across distributed brain substrates, comprising evidence to be evaluated on the extent to which stimuli in the on-going sensorimotor stream organizes distributed, parallel processing, reflecting implicit value. To effectively generate a temporally resolved phasic signal, a population of dopamine cells must exhibit synchronous activity. We argue that synchronous activity across a population of dopamine cells signals consensus across distributed afferent substrates, invigorating responding to recognized opportunities and facilitating further learning. In framing our hypothesis, we shift from the question of how value is computed to the broader question of how the brain achieves coordination across distributed, parallel processing. We posit the midbrain is part of an “axis of agency” in which the prefrontal cortex (PFC), basal ganglia (BGS), and midbrain form an axis mediating control, coordination, and consensus, respectively.

Computational models of motivated frontal function

Computational models of frontal function have made important contributions to understanding how the frontal lobes support a wide range of important functions, in their interactions with other brain areas including, critically, the basal ganglia (BG). We focus here on the specific case of how different frontal areas support goal-directed, motivated decision-making, by representing three essential types of information: possible plans of action (in more dorsal and lateral frontal areas), affectively significant outcomes of those action plans (in ventral, medial frontal areas including the orbital frontal cortex), and the overall utility of a given plan compared to other possible courses of action (in anterior cingulate cortex). Computational models of goal-directed action selection at multiple different levels of analysis provide insight into the nature of learning and processing in these areas and the relative contributions of the frontal cortex versus the BG. The most common neurologic disorders implicate these areas, and understanding their precise function and modes of dysfunction can contribute to the new field of computational psychiatry, within the broader field of computational neuroscience.

The Virtual Personalities Neural Network Model: Neurobiological Underpinnings

The Virtual Personalities Model is a motive-based neural network model that provides both a psychological model and a computational implementation that explicates the dynamics and often large within-person variability in behavior that arises over time. At the same time the same model can produce -- across many virtual personalities - between subject variability in behavior that when factor analyzed yields familiar personality structure (e.g., the Big-5). First, we describe our personality model and its implementation as a neural network model. Second, we focus on detailing the neurobiological underpinnings of this model. Third, we examine the learning mechanisms, and their biological substrates, as ways that the model gets "wired up", discussing Pavlovian and instrumental conditioning, Pavlovian to instrumental transfer (PIT), and habits. Finally, we describe the dynamics of how initial differences in propensities (e.g., dopamine functioning), wiring differences due to experience, and other factors could operate together to develop and change personality over time, and how this might be empirically examined. Thus, our goal is to contribute to the rising chorus of voices seeking a more precise neurobiologically-based science of the complex dynamics underlying personality.

A high-resolution probabilistic in vivo atlas of human subcortical brain nuclei

Recent advances in magnetic resonance imaging methods, including data acquisition, pre-processing and analysis, have benefited research on the contributions of subcortical brain nuclei to human cognition and behavior. At the same time, these developments have led to an increasing need for a high-resolution probabilistic in vivo anatomical atlas of subcortical nuclei. In order to address this need, we constructed high spatial resolution, three-dimensional templates, using high-accuracy diffeomorphic registration of T₁- and T₂- weighted structural images from 168 typical adults between 22 and 35 years old. In these templates, many tissue boundaries are clearly visible, which would otherwise be impossible to delineate in data from individual studies. The resulting delineations of subcortical nuclei complement current histology-based atlases. We further created a companion library of software tools for atlas development, to offer an open and evolving resource for the creation of a crowd-sourced in vivo probabilistic anatomical atlas of the human brain.

White noise enhances new-word learning in healthy adults

Research suggests that listening to white noise may improve some aspects of cognitive performance in individuals with lower attention. This study investigated the impact of white noise on new word learning in healthy young adults, and whether this effect was mediated by executive attention skills. Eighty participants completed a single training session to learn the names of twenty novel objects. The session comprised 5 learning phases, each followed by a recall test. A final recognition test was also administered. Half the participants listened to white noise during the learning phases, and half completed the learning in silence. The noise group demonstrated superior recall accuracy over time, which was not impacted by participant attentional capacity. Recognition accuracy was near ceiling for both groups. These findings suggest that white noise has the capacity to enhance lexical acquisition.

Lateral Hypothalamic Control of the Ventral Tegmental Area: Reward Evaluation and the Driving of Motivated Behavior

The lateral hypothalamus (LH) plays an important role in many motivated behaviors, sleep-wake states, food intake, drug-seeking, energy balance, etc. It is also home to a heterogeneous population of neurons that express and co-express multiple neuropeptides including hypocretin (Hcrt), melanin-concentrating hormone (MCH), cocaine- and amphetamine-regulated transcript (CART) and neurotensin (NT). These neurons project widely throughout the brain to areas such as the locus coeruleus, the bed nucleus of the stria terminalis, the amygdala and the ventral tegmental area (VTA). Lateral hypothalamic projections to the VTA are believed to be important for driving behavior due to the involvement of dopaminergic reward circuitry. The purpose of this article is to review current knowledge regarding the lateral hypothalamic connections to the VTA and the role they play in driving these behaviors.

The dopamine hypothesis of bipolar affective disorder: the state of the art and implications for treatment

Bipolar affective disorder is a common neuropsychiatric disorder. Although its neurobiological underpinnings are incompletely understood, the dopamine hypothesis has been a key theory of the pathophysiology of both manic and depressive phases of the illness for over four decades. The increased use of antidopaminergics in the treatment of this disorder and new in vivo neuroimaging and post-mortem studies makes it timely to review this theory. To do this, we conducted a systematic search for post-mortem, pharmacological, functional magnetic resonance and molecular imaging studies of dopamine function in bipolar disorder. Converging findings from pharmacological and imaging studies support the hypothesis that a state of hyperdopaminergia, specifically elevations in D2/3 receptor availability and a hyperactive reward processing network, underlies mania. In bipolar depression imaging studies show increased dopamine transporter levels, but changes in other aspects of dopaminergic function are inconsistent. Puzzlingly, pharmacological evidence shows that both dopamine agonists and antidopaminergics can improve bipolar depressive symptoms and perhaps actions at other receptors may reconcile these findings. Tentatively, this evidence suggests a model where an elevation in striatal D2/3 receptor availability would lead to increased dopaminergic neurotransmission and mania, whilst increased striatal dopamine transporter (DAT) levels would lead to reduced dopaminergic function and depression. Thus, it can be speculated that a failure of dopamine receptor and transporter homoeostasis might underlie the pathophysiology of this disorder. The limitations of this model include its reliance on pharmacological evidence, as these studies could potentially affect other monoamines, and the scarcity of imaging evidence on dopaminergic function. This model, if confirmed, has implications for developing new treatment strategies such as reducing the dopamine synthesis and/or release in mania and DAT blockade in bipolar depression.

Neural Circuitry of Reward Prediction Error

Dopamine neurons facilitate learning by calculating reward prediction error, or the difference between expected and actual reward. Despite two decades of research, it remains unclear how dopamine neurons make this calculation. Here we review studies that tackle this problem from a diverse set of approaches, from anatomy to electrophysiology to computational modeling and behavior. Several patterns emerge from this synthesis: that dopamine neurons themselves calculate reward prediction error, rather than inherit it passively from upstream regions; that they combine multiple separate and redundant inputs, which are themselves interconnected in a dense recurrent network; and that despite the complexity of inputs, the output from dopamine neurons is remarkably homogeneous and robust. The more we study this simple arithmetic computation, the knottier it appears to be, suggesting a daunting (but stimulating) path ahead for neuroscience more generally.

Opposite initialization to novel cues in dopamine signaling in ventral and posterior striatum in mice

Dopamine neurons are thought to encode novelty in addition to reward prediction error (the discrepancy between actual and predicted values). In this study, we compared dopamine activity across the striatum using fiber fluorometry in mice. During classical conditioning, we observed opposite dynamics in dopamine axon signals in the ventral striatum (‘VS dopamine’) and the posterior tail of the striatum (‘TS dopamine’). TS dopamine showed strong excitation to novel cues, whereas VS dopamine showed no responses to novel cues until they had been paired with a reward. TS dopamine cue responses decreased over time, depending on what the cue predicted. Additionally, TS dopamine showed excitation to several types of stimuli including rewarding, aversive, and neutral stimuli whereas VS dopamine showed excitation only to reward or reward-predicting cues. Together, these results demonstrate that dopamine novelty signals are localized in TS along with general salience signals, while VS dopamine reliably encodes reward prediction error.

DOI:http://dx.doi.org/10.7554/eLife.21886.001

New experiences trigger a variety of responses in animals. We are surprised by, move towards, and often explore new objects. But how does the brain control these responses?

Dopamine is a molecule that controls many processes in the brain and plays critical roles in various mental disorders, diseases that affect movement, and addiction. Rewarding experiences (like a glass of cold water on a hot day) can trigger dopamine neurons and studies have also shown that dopamine neurons respond to new experiences. This suggested that novelty may be rewarding in itself, or that novelty may signal the potential for future reward. On the other hand, it may be that different groups of dopamine neurons play different roles in responding to new or rewarding experiences.

In 2015, it was reported that dopamine neurons connected to the rear part of an area in the brain called the striatum receive signals from different parts of the brain than most other dopamine neurons. The dopamine neurons connected to this “tail” of the striatum preferentially received inputs from regions involved in arousal rather than reward, suggesting that they may have a unique role and transmit a different type of information.

Now, Menegas et al. have shown that dopamine signals in different areas of the striatum separate reward from novelty and other signals in mice. The results demonstrate that new odors activate dopamine neurons projecting to the tail of the striatum, but that this activity fades as the novelty wears off (as the mice learn to associate the odor with a particular outcome). By contrast, dopamine neurons projecting to the front of the striatum do not respond to novelty, but rather become more active as mice learn which odors accompany rewards (only responding to odors that predict reward). The experiments also show that dopamine neurons in the tail of the striatum encode information about the importance of a stimulus.

Together, these findings indicate that some of the roles dopamine plays in the brain may not be related to reward, but are instead linked to the novelty and importance of the stimulus. The next challenge will be to find out how the separate reward and novelty signals in dopamine neurons relate to the animals’ behavior. This may help us to better understand dopamine-related psychiatric conditions, such as depression and addiction.

DOI:http://dx.doi.org/10.7554/eLife.21886.002

3Mo: A Model for Music-Based Biofeedback

In the domain of sports and motor rehabilitation, it is of major importance to regulate and control physiological processes and physical motion in most optimal ways. For that purpose, real-time auditory feedback of physiological and physical information based on sound signals, often termed “sonification,” has been proven particularly useful. However, the use of music in biofeedback systems has been much less explored. In the current article, we assert that the use of music, and musical principles, can have a major added value, on top of mere sound signals, to the benefit of psychological and physical optimization of sports and motor rehabilitation tasks. In this article, we present the 3Mo model to describe three main functions of music that contribute to these benefits. These functions relate the power of music to Motivate, and to Monitor and Modify physiological and physical processes. The model brings together concepts and theories related to human sensorimotor interaction with music, and specifies the underlying psychological and physiological principles. This 3Mo model is intended to provide a conceptual framework that guides future research on musical biofeedback systems in the domain of sports and motor rehabilitation.

Implications of cellular models of dopamine neurons for disease

This review addresses the present state of single-cell models of the firing pattern of midbrain dopamine neurons and the insights that can be gained from these models into the underlying mechanisms for diseases such as Parkinson's, addiction, and schizophrenia. We will explain the analytical technique of separation of time scales and show how it can produce insights into mechanisms using simplified single-compartment models. We also use morphologically realistic multicompartmental models to address spatially heterogeneous aspects of neural signaling and neural metabolism. Separation of time scale analyses are applied to pacemaking, bursting, and depolarization block in dopamine neurons. Differences in subpopulations with respect to metabolic load are addressed using multicompartmental models.

Sensorimotor Grounding of Musical Embodiment and the Role of Prediction: A Review

In a previous article, we reviewed empirical evidence demonstrating action-based effects on music perception to substantiate the musical embodiment thesis (Maes et al., 2014). Evidence was largely based on studies demonstrating that music perception automatically engages motor processes, or that body states/movements influence music perception. Here, we argue that more rigorous evidence is needed before any decisive conclusion in favor of a “radical” musical embodiment thesis can be posited. In the current article, we provide a focused review of recent research to collect further evidence for the “radical” embodiment thesis that music perception is a dynamic process firmly rooted in the natural disposition of sounds and the human auditory and motor system. Though, we emphasize that, on top of these natural dispositions, long-term processes operate, rooted in repeated sensorimotor experiences and leading to learning, prediction, and error minimization. This approach sheds new light on the development of musical repertoires, and may refine our understanding of action-based effects on music perception as discussed in our previous article (Maes et al., 2014). Additionally, we discuss two of our recent empirical studies demonstrating that music performance relies on similar principles of sensorimotor dynamics and predictive processing.

Levodopa enhances explicit new-word learning in healthy adults: a preliminary study

OBJECTIVE

While the role of dopamine in modulating executive function, working memory and associative learning has been established; its role in word learning and language processing more generally is not clear. This preliminary study investigated the impact of increased synaptic dopamine levels on new-word learning ability in healthy young adults using an explicit learning paradigm.

METHOD

A double-blind, placebo-controlled, between-groups design was used. Participants completed five learning sessions over 1 week with levodopa or placebo administered at each session (five doses, 100 mg). Each session involved a study phase followed by a test phase. Test phases involved recall and recognition tests of the new (non-word) names previously paired with unfamiliar objects (half with semantic descriptions) during the study phase.

RESULTS

The levodopa group showed superior recall accuracy for new words over five learning sessions compared with the placebo group and better recognition accuracy at a 1-month follow-up for words learnt with a semantic description.

CONCLUSIONS

These findings suggest that dopamine boosts initial lexical acquisition and enhances longer-term consolidation of words learnt with semantic information, consistent with dopaminergic enhancement of semantic salience.

Arithmetic and local circuitry underlying dopamine prediction errors

Dopamine neurons are thought to facilitate learning by comparing actual and expected reward^1,2. Despite two decades of investigation, little is known about how this comparison is made. To determine how dopamine neurons calculate prediction error, we combined optogenetic manipulations with extracellular recordings in the ventral tegmental area (VTA) while mice engaged in classical conditioning. By manipulating the temporal expectation of reward, we demonstrate that dopamine neurons perform subtraction, a computation that is ideal for reinforcement learning but rarely observed in the brain. Furthermore, selectively exciting and inhibiting neighbouring GABA neurons in the VTA reveals that these neurons are a source of subtraction: they inhibit dopamine neurons when reward is expected, causally contributing to prediction error calculations. Finally, bilaterally stimulating VTA GABA neurons dramatically reduces anticipatory licking to conditioned odours, consistent with an important role for these neurons in reinforcement learning. Together, our results uncover the arithmetic and local circuitry underlying dopamine prediction errors.

Computing reward-prediction error: an integrated account of cortical timing and basal-ganglia pathways for appetitive and aversive learning

There are two prevailing notions regarding the involvement of the corticobasal ganglia system in value‐based learning: (i) the direct and indirect pathways of the basal ganglia are crucial for appetitive and aversive learning, respectively, and (ii) the activity of midbrain dopamine neurons represents reward‐prediction error. Although (ii) constitutes a critical assumption of (i), it remains elusive how (ii) holds given (i), with the basal‐ganglia influence on the dopamine neurons. Here we present a computational neural‐circuit model that potentially resolves this issue. Based on the latest analyses of the heterogeneous corticostriatal neurons and connections, our model posits that the direct and indirect pathways, respectively, represent the values of upcoming and previous actions, and up‐regulate and down‐regulate the dopamine neurons via the basal‐ganglia output nuclei. This explains how the difference between the upcoming and previous values, which constitutes the core of reward‐prediction error, is calculated. Simultaneously, it predicts that blockade of the direct/indirect pathway causes a negative/positive shift of reward‐prediction error and thereby impairs learning from positive/negative error, i.e. appetitive/aversive learning. Through simulation of reward‐reversal learning and punishment‐avoidance learning, we show that our model could indeed account for the experimentally observed features that are suggested to support notion (i) and could also provide predictions on neural activity. We also present a behavioral prediction of our model, through simulation of inter‐temporal choice, on how the balance between the two pathways relates to the subject's time preference. These results indicate that our model, incorporating the heterogeneity of the cortical influence on the basal ganglia, is expected to provide a closed‐circuit mechanistic understanding of appetitive/aversive learning.

A perspective on neural and cognitive mechanisms of error commission

Behavioral adaptation and cognitive control are crucial for goal-reaching behaviors. Every creature is ubiquitously faced with choices between behavioral alternatives. Common sense suggests that errors are an important source of information in the regulation of such processes. Several theories exist regarding cognitive control and the processing of undesired outcomes. However, most of these models focus on the consequences of an error, and less attention has been paid to the mechanisms that underlie the commissioning of an error. In this article, we present an integrative review of neuro-cognitive models that detail the determinants of the occurrence of response errors. The factors that may determine the likelihood of committing errors are likely related to the stability of task-representations in prefrontal networks, attentional selection mechanisms and mechanisms of action selection in basal ganglia circuits. An important conclusion is that the likelihood of committing an error is not stable over time but rather changes depending on the interplay of different functional neuro-anatomical and neuro-biological systems. We describe factors that might determine the time-course of cognitive control and the need to adapt behavior following response errors. Finally, we outline the mechanisms that may proof useful for predicting the outcomes of cognitive control and the emergence of response errors in future research.

A neural network model of individual differences in task switching abilities

We use a biologically grounded neural network model to investigate the brain mechanisms underlying individual differences specific to the selection and instantiation of representations that exert cognitive control in task switching. Existing computational models of task switching do not focus on individual differences and so cannot explain why task switching abilities are separable from other executive function (EF) abilities (such as response inhibition). We explore hypotheses regarding neural mechanisms underlying the "Shifting-Specific" and "Common EF" components of EF proposed in the Unity/Diversity model (Miyake & Friedman, 2012) and similar components in related theoretical frameworks. We do so by adapting a well-developed neural network model of working memory (Prefrontal cortex, Basal ganglia Working Memory or PBWM; Hazy, Frank, & O'Reilly, 2007) to task switching and the Stroop task, and comparing its behavior on those tasks under a variety of individual difference manipulations. Results are consistent with the hypotheses that variation specific to task switching (i.e., Shifting-Specific) may be related to uncontrolled, automatic persistence of goal representations, whereas variation general to multiple EFs (i.e., Common EF) may be related to the strength of PFC representations and their effect on processing in the remainder of the cognitive system. Moreover, increasing signal to noise ratio in PFC, theoretically tied to levels of tonic dopamine and a genetic polymorphism in the COMT gene, reduced Stroop interference but increased switch costs. This stability-flexibility tradeoff provides an explanation for why these two EF components sometimes show opposing correlations with other variables such as attention problems and self-restraint.

Dopamine modulates novelty seeking behavior during decision making

Novelty seeking refers to the tendency of humans and animals to explore novel and unfamiliar stimuli and environments. The idea that dopamine modulates novelty seeking is supported by evidence that novel stimuli excite dopamine neurons and activate brain regions receiving dopaminergic input. In addition, dopamine is shown to drive exploratory behavior in novel environments. It is not clear whether dopamine promotes novelty seeking when it is framed as the decision to explore novel options versus the exploitation of familiar options. To test this hypothesis, we administered systemic injections of saline or GBR-12909, a selective dopamine transporter (DAT) inhibitor, to monkeys and assessed their novelty seeking behavior during a probabilistic decision making task. The task involved pseudorandom introductions of novel choice options. This allowed monkeys the opportunity to explore novel options or to exploit familiar options that they had already sampled. We found that DAT blockade increased the monkeys' preference for novel options. A reinforcement learning (RL) model fit to the monkeys' choice data showed that increased novelty seeking after DAT blockade was driven by an increase in the initial value the monkeys assigned to novel options. However, blocking DAT did not modulate the rate at which the monkeys learned which cues were most predictive of reward or their tendency to exploit that knowledge. These data demonstrate that dopamine enhances novelty-driven value and imply that excessive novelty seeking-characteristic of impulsivity and behavioral addictions-might be caused by increases in dopamine, stemming from less reuptake.

Lasting associations between early-childhood temperament and late-adolescent reward-circuitry response to peer feedback

Behavioral inhibition, a temperament identifiable in infancy, is associated with heightened withdrawal from social encounters. Prior studies raise particular interest in the striatum, which responds uniquely to monetary gains in behaviorally inhibited children followed into adolescence. Although behavioral manifestations of inhibition are expressed primarily in the social domain, it remains unclear whether observed striatal alterations to monetary incentives also extend to social contexts. In the current study, imaging data were acquired from 39 participants (17 males, 22 females; ages 16-18 years) characterized since infancy on measures of behavioral inhibition. A social evaluation task was used to assess neural response to anticipation and receipt of positive and negative feedback from novel peers, classified by participants as being of high or low interest. As with monetary rewards, striatal response patterns differed during both anticipation and receipt of social reward between behaviorally inhibited and noninhibited adolescents. The current results, when combined with prior findings, suggest that early-life temperament predicts altered striatal response in both social and nonsocial contexts and provide support for continuity between temperament measured in early childhood and neural response to social signals measured in late adolescence and early adulthood.

Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area

Neural activity in dopaminergic areas such as the ventral tegmental area is influenced by timing processes, in particular by the temporal expectation of rewards during Pavlovian conditioning. Receipt of a reward at the expected time allows to compute reward-prediction errors which can drive learning in motor or cognitive structures. Reciprocally, dopamine plays an important role in the timing of external events. Several models of the dopaminergic system exist, but the substrate of temporal learning is rather unclear. In this article, we propose a neuro-computational model of the afferent network to the ventral tegmental area, including the lateral hypothalamus, the pedunculopontine nucleus, the amygdala, the ventromedial prefrontal cortex, the ventral basal ganglia (including the nucleus accumbens and the ventral pallidum), as well as the lateral habenula and the rostromedial tegmental nucleus. Based on a plausible connectivity and realistic learning rules, this neuro-computational model reproduces several experimental observations, such as the progressive cancelation of dopaminergic bursts at reward delivery, the appearance of bursts at the onset of reward-predicting cues or the influence of reward magnitude on activity in the amygdala and ventral tegmental area. While associative learning occurs primarily in the amygdala, learning of the temporal relationship between the cue and the associated reward is implemented as a dopamine-modulated coincidence detection mechanism in the nucleus accumbens.

Multiplexing signals in reinforcement learning with internal models and dopamine

A fundamental challenge for computational and cognitive neuroscience is to understand how reward-based learning and decision-making are made and how accrued knowledge and internal models of the environment are incorporated. Remarkable progress has been made in the field, guided by the midbrain dopamine reward prediction error hypothesis and the underlying reinforcement learning framework, which does not involve internal models ('model-free'). Recent studies, however, have begun not only to address more complex decision-making processes that are integrated with model-free decision-making, but also to include internal models about environmental reward structures and the minds of other agents, including model-based reinforcement learning and using generalized prediction errors. Even dopamine, a classic model-free signal, may work as multiplexed signals using model-based information and contribute to representational learning of reward structure.

Interactions of timing and prediction error learning

Timing and prediction error learning have historically been treated as independent processes, but growing evidence has indicated that they are not orthogonal. Timing emerges at the earliest time point when conditioned responses are observed, and temporal variables modulate prediction error learning in both simple conditioning and cue competition paradigms. In addition, prediction errors, through changes in reward magnitude or value alter timing of behavior. Thus, there appears to be a bi-directional interaction between timing and prediction error learning. Modern theories have attempted to integrate the two processes with mixed success. A neurocomputational approach to theory development is espoused, which draws on neurobiological evidence to guide and constrain computational model development. Heuristics for future model development are presented with the goal of sparking new approaches to theory development in the timing and prediction error fields.

Strategic cognitive sequencing: a computational cognitive neuroscience approach

We address strategic cognitive sequencing, the "outer loop" of human cognition: how the brain decides what cognitive process to apply at a given moment to solve complex, multistep cognitive tasks. We argue that this topic has been neglected relative to its importance for systematic reasons but that recent work on how individual brain systems accomplish their computations has set the stage for productively addressing how brain regions coordinate over time to accomplish our most impressive thinking. We present four preliminary neural network models. The first addresses how the prefrontal cortex (PFC) and basal ganglia (BG) cooperate to perform trial-and-error learning of short sequences; the next, how several areas of PFC learn to make predictions of likely reward, and how this contributes to the BG making decisions at the level of strategies. The third models address how PFC, BG, parietal cortex, and hippocampus can work together to memorize sequences of cognitive actions from instruction (or "self-instruction"). The last shows how a constraint satisfaction process can find useful plans. The PFC maintains current and goal states and associates from both of these to find a "bridging" state, an abstract plan. We discuss how these processes could work together to produce strategic cognitive sequencing and discuss future directions in this area.

The detection of novelty relies on dopaminergic signaling: evidence from apomorphine's impact on the novelty N2

Despite much research, it remains unclear if dopamine is directly involved in novelty detection or plays a role in orchestrating the subsequent cognitive response. This ambiguity stems in part from a reliance on experimental designs where novelty is manipulated and dopaminergic activity is subsequently observed. Here we adopt the alternative approach: we manipulate dopamine activity using apomorphine (D1/D2 agonist) and measure the change in neurological indices of novelty processing. In separate drug and placebo sessions, participants completed a von Restorff task. Apomorphine speeded and potentiated the novelty-elicited N2, an Event-Related Potential (ERP) component thought to index early aspects of novelty detection, and caused novel-font words to be better recalled. Apomorphine also decreased the amplitude of the novelty-P3a. An increase in D1/D2 receptor activation thus appears to potentiate neural sensitivity to novel stimuli, causing this content to be better encoded.

The parietal cortex in sensemaking: the dissociation of multiple types of spatial information

According to the data-frame theory, sensemaking is a macrocognitive process in which people try to make sense of or explain their observations by processing a number of explanatory structures called frames until the observations and frames become congruent. During the sensemaking process, the parietal cortex has been implicated in various cognitive tasks for the functions related to spatial and temporal information processing, mathematical thinking, and spatial attention. In particular, the parietal cortex plays important roles by extracting multiple representations of magnitudes at the early stages of perceptual analysis. By a series of neural network simulations, we demonstrate that the dissociation of different types of spatial information can start early with a rather similar structure (i.e., sensitivity on a common metric), but accurate representations require specific goal-directed top-down controls due to the interference in selective attention. Our results suggest that the roles of the parietal cortex rely on the hierarchical organization of multiple spatial representations and their interactions. The dissociation and interference between different types of spatial information are essentially the result of the competition at different levels of abstraction.

A biologically plausible embodied model of action discovery

During development, animals can spontaneously discover action-outcome pairings enabling subsequent achievement of their goals. We present a biologically plausible embodied model addressing key aspects of this process. The biomimetic model core comprises the basal ganglia and its loops through cortex and thalamus. We incorporate reinforcement learning (RL) with phasic dopamine supplying a sensory prediction error, signalling "surprising" outcomes. Phasic dopamine is used in a cortico-striatal learning rule which is consistent with recent data. We also hypothesized that objects associated with surprising outcomes acquire "novelty salience" contingent on the predicability of the outcome. To test this idea we used a simple model of prediction governing the dynamics of novelty salience and phasic dopamine. The task of the virtual robotic agent mimicked an in vivo counterpart (Gancarz et al., 2011) and involved interaction with a target object which caused a light flash, or a control object which did not. Learning took place according to two schedules. In one, the phasic outcome was delivered after interaction with the target in an unpredictable way which emulated the in vivo protocol. Without novelty salience, the model was unable to account for the experimental data. In the other schedule, the phasic outcome was reliably delivered and the agent showed a rapid increase in the number of interactions with the target which then decreased over subsequent sessions. We argue this is precisely the kind of change in behavior required to repeatedly present representations of context, action and outcome, to neural networks responsible for learning action-outcome contingency. The model also showed cortico-striatal plasticity consistent with learning a new action in basal ganglia. We conclude that action learning is underpinned by a complex interplay of plasticity and stimulus salience, and that our model contains many of the elements for biological action discovery to take place.

Assembling old tricks for new tasks: a neural model of instructional learning and control

We can learn from the wisdom of others to maximize success. However, it is unclear how humans take advice to flexibly adapt behavior. On the basis of data from neuroanatomy, neurophysiology, and neuroimaging, a biologically plausible model is developed to illustrate the neural mechanisms of learning from instructions. The model consists of two complementary learning pathways. The slow-learning parietal pathway carries out simple or habitual stimulus-response (S-R) mappings, whereas the fast-learning hippocampal pathway implements novel S-R rules. Specifically, the hippocampus can rapidly encode arbitrary S-R associations, and stimulus-cued responses are later recalled into the basal ganglia-gated pFC to bias response selection in the premotor and motor cortices. The interactions between the two model learning pathways explain how instructions can override habits and how automaticity can be achieved through motor consolidation.

A Spiking Neural Model for Stable Reinforcement of Synapses Based on Multiple Distal Rewards

In this letter, a novel critic-like algorithm was developed to extend the synaptic plasticity rule described in Florian ( 2007 ) and Izhikevich ( 2007 ) in order to solve the problem of learning multiple distal rewards simultaneously. The system is augmented with short-term plasticity (STP) to stabilize the learning dynamics, thereby increasing the system's learning capacity. A theoretical threshold is estimated for the number of distal rewards that this system can learn. The validity of the novel algorithm was verified by computer simulations.

Dopaminergic involvement during mental fatigue in health and cocaine addiction

Dopamine modulates executive function, including sustaining cognitive control during mental fatigue. Using event-related functional magnetic resonance imaging (fMRI) during the color-word Stroop task, we aimed to model mental fatigue with repeated task exposures in 33 cocaine abusers and 20 healthy controls. During such mental fatigue (indicated by increased errors, and decreased post-error slowing and dorsal anterior cingulate response to error as a function of time-on-task), healthy individuals showed increased activity in the dopaminergic midbrain to error. Cocaine abusers, characterized by disrupted dopamine neurotransmission, showed an opposite pattern of response. This midbrain fMRI activity with repetition was further correlated with objective indices of endogenous motivation in all subjects: a state measure (task reaction time) and a trait measure (dopamine D2 receptor availability in caudate, as revealed by positron emission tomography data collected in a subset of this sample, which directly points to a contribution of dopamine to these results). In a second sample of 14 cocaine abusers and 15 controls, administration of an indirect dopamine agonist, methylphenidate, reversed these midbrain responses in both groups, possibly indicating normalization of response in cocaine abusers because of restoration of dopamine signaling but degradation of response in healthy controls owing to excessive dopamine signaling. Together, these multimodal imaging findings suggest a novel involvement of the dopaminergic midbrain in sustaining motivation during fatigue. This region might provide a useful target for strengthening self-control and/or endogenous motivation in addiction.

Anorexia nervosa and obesity are associated with opposite brain reward response

Anorexia nervosa (AN) is a severe psychiatric disorder associated with food avoidance and malnutrition. In this study, we wanted to test whether we would find brain reward alterations in AN, compared with individuals with normal or increased body weight. We studied 21 underweight, restricting-type AN (age M 22.5, SD 5.8 years), 19 obese (age M 27.1, SD 6.7 years), and 23 healthy control women (age M 24.8, SD 5.6 years), using blood oxygen level-dependent functional magnetic resonance brain imaging together with a reward-conditioning task. This paradigm involves learning the association between conditioned visual stimuli and unconditioned taste stimuli, as well as the unexpected violation of those learned associations. The task has been associated with activation of brain dopamine reward circuits, and it allows the comparison of actual brain response with expected brain activation based on established neuronal models. A group-by-task condition analysis (family-wise-error-corrected P<0.05) indicated that the orbitofrontal cortex differentiated all three groups. The dopamine model reward-learning signal distinguished groups in the anteroventral striatum, insula, and prefrontal cortex (P<0.001, 25 voxel cluster threshold), with brain responses that were greater in the AN group, but lesser in the obese group, compared with controls. These results suggest that brain reward circuits are more responsive to food stimuli in AN, but less responsive in obese women. The mechanism for this association is uncertain, but these brain reward response patterns could be biomarkers for the respective weight state.

Complementary neural correlates of motivation in dopaminergic and noradrenergic neurons of monkeys

Rewards have many influences on learning, decision-making, and performance. All seem to rely on complementary actions of two closely related catecholaminergic neuromodulators, dopamine (DA), and noradrenaline (NA). We compared single unit activity of dopaminergic neurons of the substantia nigra pars compacta (SNc) and noradrenergic neurons of the locus coeruleus (LC) in monkeys performing a reward schedule task. Their motivation, indexed using operant performance, increased as they progressed through schedules ending in reward delivery. The responses of dopaminergic and noradrenergic neurons around the time of major task events, visual cues predicting trial outcome and operant action to complete a trial were similar in that they occurred at the same time. They were also similar in that they both responded most strongly to the first cues in schedules, which are the most informative cues. The neuronal responses around the time of the monkeys' actions were different, in that the response intensity profiles changed in opposite directions. Dopaminergic responses were stronger around predictably rewarded correct actions whereas noradrenergic responses were greater around predictably unrewarded correct actions. The complementary response profiles related to the monkeys operant actions suggest that DA neurons might relate to the value of the current action whereas the noradrenergic neurons relate to the psychological cost of that action.

Understanding interpersonal function in psychiatric illness through multiplayer economic games

Interpersonal factors play significant roles in the onset, maintenance, and remission of psychiatric conditions. In the current major diagnostic classification systems for psychiatric disorders, some conditions are defined by the presence of impairments in social interaction or maintaining interpersonal relationships; these include autism, social phobia, and the personality disorders. Other psychopathologies confer significant difficulties in the social domain, including major depression, posttraumatic stress disorder, and psychotic disorders. Still other mental health conditions, including substance abuse and eating disorders, seem to be exacerbated or triggered in part by the influence of social peers. For each of these and other psychiatric conditions, the extent and quality of social support is a strong determinant of outcome such that high social support predicts symptom improvement and remission. Despite the central role of interpersonal factors in psychiatric illness, the neurobiology of social impairments remains largely unexplored, in part due to difficulties eliciting and quantifying interpersonal processes in a parametric manner. Recent advances in functional neuroimaging, combined with multiplayer exchange games drawn from behavioral economics, and computational/quantitative approaches more generally, provide a fitting paradigm within which to study interpersonal function and dysfunction in psychiatric conditions. In this review, we outline the importance of interpersonal factors in psychiatric illness and discuss ways in which neuroeconomics provides a tractable framework within which to examine the neurobiology of social dysfunction.

Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis

Growing evidence suggests that the prefrontal cortex (PFC) is organized hierarchically, with more anterior regions having increasingly abstract representations. How does this organization support hierarchical cognitive control and the rapid discovery of abstract action rules? We present computational models at different levels of description. A neural circuit model simulates interacting corticostriatal circuits organized hierarchically. In each circuit, the basal ganglia gate frontal actions, with some striatal units gating the inputs to PFC and others gating the outputs to influence response selection. Learning at all of these levels is accomplished via dopaminergic reward prediction error signals in each corticostriatal circuit. This functionality allows the system to exhibit conditional if-then hypothesis testing and to learn rapidly in environments with hierarchical structure. We also develop a hybrid Bayesian-reinforcement learning mixture of experts (MoE) model, which can estimate the most likely hypothesis state of individual participants based on their observed sequence of choices and rewards. This model yields accurate probabilistic estimates about which hypotheses are attended by manipulating attentional states in the generative neural model and recovering them with the MoE model. This 2-pronged modeling approach leads to multiple quantitative predictions that are tested with functional magnetic resonance imaging in the companion paper.

Synchrony of corticostriatal-midbrain activation enables normal inhibitory control and conflict processing in recovering alcoholic men

BACKGROUND

Alcohol dependence is associated with inhibitory control deficits, possibly related to abnormalities in frontoparietal cortical and midbrain function and connectivity.

METHODS

We examined functional connectivity and microstructural fiber integrity between frontoparietal and midbrain structures using a Stroop Match-to-Sample task with functional magnetic resonance imaging and diffusion tensor imaging in 18 alcoholic and 17 control subjects. Manipulation of color cues and response repetition sequences modulated cognitive demands during Stroop conflict.

RESULTS

Despite similar lateral frontoparietal activity and functional connectivity in alcoholic and control subjects when processing conflict, control subjects deactivated the posterior cingulate cortex (PCC), whereas alcoholic subjects did not. Posterior cingulum fiber integrity predicted the degree of PCC deactivation in control but not alcoholic subjects. Also, PCC activity was modulated by executive control demands: activated during response switching and deactivated during response repetition. Alcoholics showed the opposite pattern: activation during repetition and deactivation during switching. Here, in alcoholic subjects, greater deviations from the normal PCC activity correlated with higher amounts of lifetime alcohol consumption. A functional dissociation of brain network connectivity between the groups further showed that control subjects exhibited greater corticocortical connectivity among middle cingulate, posterior cingulate, and medial prefrontal cortices than alcoholic subjects. In contrast, alcoholic subjects exhibited greater midbrain-orbitofrontal cortical network connectivity than control subjects. Degree of microstructural fiber integrity predicted robustness of functional connectivity.

CONCLUSIONS

Thus, even subtle compromise of microstructural connectivity in alcoholism can influence modulation of functional connectivity and underlie alcohol-related cognitive impairment.

Dopamine, affordance and active inference

The role of dopamine in behaviour and decision-making is often cast in terms of reinforcement learning and optimal decision theory. Here, we present an alternative view that frames the physiology of dopamine in terms of Bayes-optimal behaviour. In this account, dopamine controls the precision or salience of (external or internal) cues that engender action. In other words, dopamine balances bottom-up sensory information and top-down prior beliefs when making hierarchical inferences (predictions) about cues that have affordance. In this paper, we focus on the consequences of changing tonic levels of dopamine firing using simulations of cued sequential movements. Crucially, the predictions driving movements are based upon a hierarchical generative model that infers the context in which movements are made. This means that we can confuse agents by changing the context (order) in which cues are presented. These simulations provide a (Bayes-optimal) model of contextual uncertainty and set switching that can be quantified in terms of behavioural and electrophysiological responses. Furthermore, one can simulate dopaminergic lesions (by changing the precision of prediction errors) to produce pathological behaviours that are reminiscent of those seen in neurological disorders such as Parkinson's disease. We use these simulations to demonstrate how a single functional role for dopamine at the synaptic level can manifest in different ways at the behavioural level.

The neurobiology of anhedonia and other reward-related deficits

Anhedonia, or markedly diminished interest or pleasure, is a hallmark symptom of major depression, schizophrenia and other neuropsychiatric disorders. Over the past three decades, the clinical definition of anhedonia has remained relatively unchanged, although cognitive psychology and behavioral neuroscience have expanded our understanding of other reward-related processes. Here, we review the neural bases of the construct of anhedonia that reflects deficits in hedonic capacity and also closely linked to the constructs of reward valuation, decision-making, anticipation and motivation. The neural circuits subserving these reward-related processes include the ventral striatum, prefrontal cortical regions, and afferent and efferent projections. An understanding of anhedonia and other reward-related constructs will facilitate the diagnosis and treatment of disorders that include reward deficits as key symptoms.