Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area

Focal mu-opioid receptor activation promotes neuroinflammation and microglial activation in the mesocorticolimbic system: Alterations induced by inflammatory pain

Microglia participates in the modulation of pain signaling. The activation of microglia is suggested to play an important role in affective disorders that are related to a dysfunction of the mesocorticolimbic system (MCLS) and are commonly associated with chronic pain. Moreover, there is evidence that mu-opioid receptors (MORs), expressed in the MCLS, are involved in neuroinflammatory events, although the way by which they do it remains to be elucidated. In this study, we propose that MOR pharmacological activation within the MCLS activates and triggers the local release of proinflammatory cytokines and this pattern of activation is impacted by the presence of systemic inflammatory pain. To test this hypothesis, we used in vivo microdialysis coupled with flow cytometry to measure cytokines release in the nucleus accumbens and immunofluorescence of IBA1 in areas of the MCLS on a rat model of inflammatory pain. Interestingly, the treatment with DAMGO, a MOR agonist locally in the nucleus accumbens, triggered the release of the IL1α, IL1β, and IL6 proinflammatory cytokines. Furthermore, MOR pharmacological activation in the ventral tegmental area (VTA) modified the levels of IBA1-positive cells in the VTA, prefrontal cortex, the nucleus accumbens and the amygdala in a dose-dependent way, without impacting mechanical nociception. Additionally, MOR blockade in the VTA prevents DAMGO-induced effects. Finally, we observed that systemic inflammatory pain altered the IBA1 immunostaining derived from MOR activation in the MSCLS. Altogether, our results indicate that the microglia-MOR relationship could be pivotal to unravel some inflammatory pain-induced comorbidities related to MCLS dysfunction.

A VTA GABAergic computational model of dissociated reward prediction error computation in classical conditioning

Classical Conditioning is a fundamental learning mechanism where the Ventral Striatum is generally thought to be the source of inhibition to Ventral Tegmental Area (VTA) Dopamine neurons when a reward is expected. However, recent evidences point to a new candidate in VTA GABA encoding expectation for computing the reward prediction error in the VTA. In this system-level computational model, the VTA GABA signal is hypothesised to be a combination of magnitude and timing computed in the Peduncolopontine and Ventral Striatum respectively. This dissociation enables the model to explain recent results wherein Ventral Striatum lesions affected the temporal expectation of the reward but the magnitude of the reward was intact. This model also exhibits other features in classical conditioning namely, progressively decreasing firing for early rewards closer to the actual reward, twin peaks of VTA dopamine during training and cancellation of US dopamine after training.

The Role of the Lateral Habenula in Inhibitory Learning from Reward Omission

The lateral habenula (LHb) is a phylogenetically primitive brain structure that plays a key role in learning to inhibit distinct responses to specific stimuli. This structure is activated by primary aversive stimuli, cues predicting an imminent aversive event, unexpected reward omissions, and cues associated with the omission of an expected reward. The most widely described physiological effect of LHb activation is acutely suppressing midbrain dopaminergic signaling. However, recent studies have identified multiple means by which the LHb promotes this effect as well as other mechanisms of action. These findings reveal the complex nature of LHb circuitry. The present paper reviews the role of this structure in learning from reward omission. We approach this topic from the perspective of computational models of behavioral change that account for inhibitory learning to frame key findings. Such findings are drawn from recent behavioral neuroscience studies that use novel brain imaging, stimulation, ablation, and reversible inactivation techniques. Further research and conceptual work are needed to clarify the nature of the mechanisms related to updating motivated behavior in which the LHb is involved. As yet, there is little understanding of whether such mechanisms are parallel or complementary to the well-known modulatory function of the more recently evolved prefrontal cortex.

Modulation of Dopamine for Adaptive Learning: A Neurocomputational Model

There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine mediated tasks that transcend the experimental phenomena it was initially designed to address.

A systems-neuroscience model of phasic dopamine

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Computational modeling of the monoaminergic neurotransmitter and male neuroendocrine systems in an analysis of therapeutic neuroadaptation to chronic antidepressant

Second-line depression treatment involves augmentation with one (rarely two) additional drugs, of chronic administration of a selective serotonin reuptake inhibitor (SSRI), which is the first-line depression treatment. Unfortunately, many depressed patients still fail to respond even after months to years of searching to find an effective combination. To aid in the identification of potentially effective antidepressant combinations, we created a computational model of the monoaminergic neurotransmitter (serotonin, norepinephrine, and dopamine), stress-hormone (cortisol), and male sex hormone (testosterone) systems. The model was trained via machine learning to represent a broad range of empirical observations. Neuroadaptation to chronic drug administration was simulated through incremental adjustments in model parameters that corresponded to key regulatory components of the neurotransmitter and neurohormone systems. Analysis revealed that neuroadaptation in the model depended on all of the regulatory components in complicated ways, and did not reveal any one or a few specific components that could be targeted in the design of antidepressant treatments. We used large sets of neuroadapted states of the model to screen 74 different drug and hormone combinations and identified several combinations that could potentially be therapeutic for a higher proportion of male patients than SSRIs by themselves.

Synchronicity: The Role of Midbrain Dopamine in Whole-Brain Coordination

Midbrain dopamine seems to play an outsized role in motivated behavior and learning. Widely associated with mediating reward-related behavior, decision making, and learning, dopamine continues to generate controversies in the field. While many studies and theories focus on what dopamine cells encode, the question of how the midbrain derives the information it encodes is poorly understood and comparatively less addressed. Recent anatomical studies suggest greater diversity and complexity of afferent inputs than previously appreciated, requiring rethinking of prior models. Here, we elaborate a hypothesis that construes midbrain dopamine as implementing a Bayesian selector in which individual dopamine cells sample afferent activity across distributed brain substrates, comprising evidence to be evaluated on the extent to which stimuli in the on-going sensorimotor stream organizes distributed, parallel processing, reflecting implicit value. To effectively generate a temporally resolved phasic signal, a population of dopamine cells must exhibit synchronous activity. We argue that synchronous activity across a population of dopamine cells signals consensus across distributed afferent substrates, invigorating responding to recognized opportunities and facilitating further learning. In framing our hypothesis, we shift from the question of how value is computed to the broader question of how the brain achieves coordination across distributed, parallel processing. We posit the midbrain is part of an “axis of agency” in which the prefrontal cortex (PFC), basal ganglia (BGS), and midbrain form an axis mediating control, coordination, and consensus, respectively.

Ventral tegmental area deep brain stimulation for chronic cluster headache: Effects on cognition, mood, pain report behaviour and quality of life

Background

Deep brain stimulation in the ventral tegmental area (VTA-DBS) has provided remarkable therapeutic benefits in decreasing headache frequency and severity in patients with medically refractory chronic cluster headache (CH). However, to date the effects of VTA-DBS on cognition, mood and quality of life have not been examined in detail.

Methods

The aim of the present study was to do so in a case series of 18 consecutive patients with cluster headache who underwent implantation of deep brain stimulation electrodes in the ventral tegmental area. The patients were evaluated preoperatively and after a mean of 14 months of VTA-DBS on tests of global cognition (Mini Mental State Examination), intelligence (Wechsler Abbreviated Scale of Intelligence), verbal memory (California Verbal Learning Test-II), executive function (Delis–Kaplan Executive Function System), and attention (Paced Auditory Serial Addition Test). Depression (Beck Depression Inventory and Hospital Anxiety and Depression Rating Scale-D), anxiety (Hospital Anxiety and Depression Rating Scale-A), apathy (Starkstein Apathy Scale), and hopelessness (Beck Hopelessness Scale) were also assessed. Subjective pain experience (McGill Pain Questionnaire), behaviour (Pain Behaviour Checklist) and quality of life (Short Form-36) were also evaluated at the same time points.

Results

VTA-DBS resulted in significant improvement of headache frequency (from a mean of five to two attacks daily, p < .001) and severity (from mean Verbal Rating Scale [VRS] of 10 to 7, p < .001) which was associated with significant reduction of anxiety (from mean HADS-A of 11.94 to 8.00, p < .001) and help-seeking behaviours (from mean PBC of 4.00 to 2.61, p < .001). VTA-DBS did not produce any significant change to any tests of cognitive function and any other outcome measures (BDI, HADS-D, SAS, BHS, McGill Pain Questionnaire, Short Form-36).

Conclusion

We confirm the efficacy of VTA-DBS in the treatment of medically refractory chronic cluster headache. The reduction of headache frequency and severity was associated with a significant reduction of anxiety. Furthermore, the result suggests that VTA-DBS for chronic cluster headache improves pain-related help-seeking behaviours and does not produce any change in cognition.

Minimal Circuit Model of Reward Prediction Error Computations and Effects of Nicotinic Modulations

Dopamine (DA) neurons in the ventral tegmental area (VTA) are thought to encode reward prediction errors (RPE) by comparing actual and expected rewards. In recent years, much work has been done to identify how the brain uses and computes this signal. While several lines of evidence suggest the interplay of the DA and the inhibitory interneurons in the VTA implements the RPE computation, it still remains unclear how the DA neurons learn key quantities, for example the amplitude and the timing of primary rewards during conditioning tasks. Furthermore, endogenous acetylcholine and exogenous nicotine, also likely affect these computations by acting on both VTA DA and GABA (γ -aminobutyric acid) neurons via nicotinic-acetylcholine receptors (nAChRs). To explore the potential circuit-level mechanisms for RPE computations during classical-conditioning tasks, we developed a minimal computational model of the VTA circuitry. The model was designed to account for several reward-related properties of VTA afferents and recent findings on VTA GABA neuron dynamics during conditioning. With our minimal model, we showed that the RPE can be learned by a two-speed process computing reward timing and magnitude. By including models of nAChR-mediated currents in the VTA DA-GABA circuit, we showed that nicotine should reduce the acetylcholine action on the VTA GABA neurons by receptor desensitization and potentially boost DA responses to reward-related signals in a non-trivial manner. Together, our results delineate the mechanisms by which RPE are computed in the brain, and suggest a hypothesis on nicotine-mediated effects on reward-related perception and decision-making.

On the Role of Cortex-Basal Ganglia Interactions for Category Learning: A Neurocomputational Approach

In addition to the prefrontal cortex (PFC), the basal ganglia (BG) have been increasingly often reported to play a fundamental role in category learning, but the circuit mechanisms mediating their interaction remain to be explored. We developed a novel neurocomputational model of category learning that particularly addresses the BG-PFC interplay. We propose that the BG bias PFC activity by removing the inhibition of cortico-thalamo-cortical loop and thereby provide a teaching signal to guide the acquisition of category representations in the corticocortical associations to the PFC. Our model replicates key behavioral and physiological data of macaque monkey learning a prototype distortion task from Antzoulatos and Miller (2011) Our simulations allowed us to gain a deeper insight into the observed drop of category selectivity in striatal neurons seen in the experimental data and in the model. The simulation results and a new analysis of the experimental data based on the model's predictions show that the drop in category selectivity of the striatum emerges as the variability of responses in the striatum rises when confronting the BG with an increasingly larger number of stimuli to be classified. The neurocomputational model therefore provides new testable insights of systems-level brain circuits involved in category learning that may also be generalized to better understand other cortico-BG-cortical loops.SIGNIFICANCE STATEMENT Inspired by the idea that basal ganglia (BG) teach the prefrontal cortex (PFC) to acquire category representations, we developed a novel neurocomputational model and tested it on a task that was recently applied in monkey experiments. As an advantage over previous models of category learning, our model allows to compare simulation data with single-cell recordings in PFC and BG. We not only derived model predictions, but already verified a prediction to explain the observed drop in striatal category selectivity. When testing our model with a simple, real-world face categorization task, we observed that the fast striatal learning with a performance of 85% correct responses can teach the slower PFC learning to push the model performance up to almost 100%.

Differential release of dopamine in the nucleus accumbens evoked by low-versus high-frequency medial prefrontal cortex stimulation

The medial prefrontal cortex (mPFC) coordinates goal-directed behaviors, which may be mediated through mPFC regulation of dopamine release in the nucleus accumbens (NAc). Furthermore, frequency-specific oscillatory activity between the frontal cortex and downstream structures may facilitate inter-region communication. Although high-frequency (e.g., 60 Hz) mPFC stimulation is known to increase basal dopamine levels in the NAc, little is known about how phasic dopamine release is affected by mPFC stimulation. Understanding the frequency-specific control of phasic dopamine release by mPFC stimulation could elucidate mechanisms by which the mPFC modulates other regions. It could also inform optimization of deep brain stimulation for treatment of neurological disorders.

OBJECTIVE

The goal of this work was to characterize the frequency response of NAc dopamine release resultant from mPFC stimulation. We hypothesized that the magnitude of dopamine release in the NAc would increase with increasing stimulation frequency.

METHODS

Electrical stimulation of the mPFC of anesthetized rats was delivered at 4-60 Hz and at varying durations while measuring NAc dopamine release with fast-scan cyclic voltammetry.

RESULTS

mPFC stimulation resulted in phasic dopamine release in the NAc. Furthermore, 20 Hz stimulation evoked the largest peak response for stimulation intervals >5 s when compared to higher or lower frequencies.

CONCLUSIONS

Activation of the mPFC drives dopamine release in the NAc in a complex frequency- and duration-dependent manner. This has implications for the use of deep brain stimulation treatment of disorders marked by dopaminergic dysregulation, and suggest that mPFC may exert more specialized control over neuromodulator release than previously understood.

Neural Circuitry of Reward Prediction Error

Dopamine neurons facilitate learning by calculating reward prediction error, or the difference between expected and actual reward. Despite two decades of research, it remains unclear how dopamine neurons make this calculation. Here we review studies that tackle this problem from a diverse set of approaches, from anatomy to electrophysiology to computational modeling and behavior. Several patterns emerge from this synthesis: that dopamine neurons themselves calculate reward prediction error, rather than inherit it passively from upstream regions; that they combine multiple separate and redundant inputs, which are themselves interconnected in a dense recurrent network; and that despite the complexity of inputs, the output from dopamine neurons is remarkably homogeneous and robust. The more we study this simple arithmetic computation, the knottier it appears to be, suggesting a daunting (but stimulating) path ahead for neuroscience more generally.

Arithmetic and local circuitry underlying dopamine prediction errors

Dopamine neurons are thought to facilitate learning by comparing actual and expected reward^1,2. Despite two decades of investigation, little is known about how this comparison is made. To determine how dopamine neurons calculate prediction error, we combined optogenetic manipulations with extracellular recordings in the ventral tegmental area (VTA) while mice engaged in classical conditioning. By manipulating the temporal expectation of reward, we demonstrate that dopamine neurons perform subtraction, a computation that is ideal for reinforcement learning but rarely observed in the brain. Furthermore, selectively exciting and inhibiting neighbouring GABA neurons in the VTA reveals that these neurons are a source of subtraction: they inhibit dopamine neurons when reward is expected, causally contributing to prediction error calculations. Finally, bilaterally stimulating VTA GABA neurons dramatically reduces anticipatory licking to conditioned odours, consistent with an important role for these neurons in reinforcement learning. Together, our results uncover the arithmetic and local circuitry underlying dopamine prediction errors.

The strategy and motivational influences on the beneficial effect of neurostimulation: a tDCS and fNIRS study

Working memory (WM) capacity falls along a spectrum with some people demonstrating higher and others lower WM capacity. Efforts to improve WM include applying transcranial direct current stimulation (tDCS), in which small amounts of current modulate the activity of underlying neurons and enhance cognitive function. However, not everyone benefits equally from a given tDCS protocol. Recent findings revealed tDCS-related WM benefits for individuals with higher working memory (WM) capacity. Here, we test two hypotheses regarding those with low WM capacity to see if they too would benefit under more optimal conditions. We tested whether supplying a WM strategy (Experiment 1) or providing greater extrinsic motivation through incentives (Experiment 2) would restore tDCS benefit to the low WM capacity group. We also employed functional near infrared spectroscopy to monitor tDCS-induced changes in neural activity. Experiment 1 demonstrated that supplying a WM strategy improved the high WM capacity participants' accuracy and the amount of oxygenated blood levels following anodal tDCS, but it did not restore tDCS-linked WM benefits to the low WM capacity group. Experiment 2 demonstrated that financial motivation enhanced performance in both low and high WM capacity groups, especially after anodal tDCS. Here, only the low WM capacity participants showed a generalized increase in oxygenated blood flow across both low and high motivation conditions. These results indicate that ensuring that participants' incentives are high may expand cognitive benefits associated with tDCS. This finding is relevant for translational work using tDCS in clinical populations, in which motivation can be a concern.

Endocannabinoid-dependent modulation of phasic dopamine signaling encodes external and internal reward-predictive cues

The mesolimbic dopamine (DA) system plays an integral role in incentive motivation and reward seeking and a growing body of evidence identifies signal transduction at cannabinoid receptors as a critical modulator of this system. Indeed, administration of exogenous cannabinoids results in burst firing of DA neurons of the ventral tegmental area and increases extracellular DA in the nucleus accumbens (NAcc). Implementation of fast-scan cyclic voltammetry (FSCV) confirms the ability of cannabinoids to augment DA within the NAcc on a subsecond timescale. The use of FSCV along with newly developed highly selective pharmacological compounds advances our understanding of how cannabinoids influence DA transmission and highlights a role for endocannabinoid-modulated subsecond DAergic activation in the incentive motivational properties of not only external, but also internal reward-predictive cues. For example, our laboratory has recently demonstrated that in mice responding under a fixed-interval (FI) schedule for food reinforcement, fluctuations in NAcc DA signal the principal cue predictive of reinforcer availability - time. That is, as the interval progresses, NAcc DA levels decline leading to accelerated food seeking and the resulting characteristic FI scallop pattern of responding. Importantly, administration of WIN 55,212-2, a synthetic cannabinoid agonist, or JZL184, an indirect cannabinoid agonist, increases DA levels during the interval and disrupts this pattern of responding. Along with a wealth of other reports, these results illustrate the role of cannabinoid receptor activation in the regulation of DA transmission and the control of temporally guided reward seeking. The current review will explore the striatal beat frequency model of interval timing as it pertains to cannabinoid signaling and propose a neurocircuitry through which this system modulates interoceptive time cues.