Activation of dorsal raphe serotonin neurons underlies waiting for delayed rewards

Association with reward negatively modulates short latency phasic conditioned responses of dorsal raphe nucleus neurons in freely moving rats

The dorsal raphe nucleus (DRN) is implicated in mood regulation, control of impulsive behavior, and in processing aversive and reward-related signals. DRN neurons show phasic responses to sensory stimuli, but whether association with reward modulates these responses is unknown. We recorded DRN neurons from rats in a contextual conditioned approach paradigm in which an auditory cue was either followed or not followed by reward, depending on a global context signal. Conditioned approach (licking) occurred after cues in the reward context, but was suppressed in the no-reward context. Many DRN neurons showed short-latency phasic activations in response to the cues. There was striking contextual bias, with more and stronger excitations in the no-reward context than in the reward context. Therefore, DRN activity scaled inversely with cue salience and with the probability of subsequent conditioned approach. Tonic changes were similarly discriminatory, with increases being dominant after cues in the no-reward context, when licking was suppressed, and tonic decreases in rate dominant after reward-predictive cues during expression of conditioned licking. Phasic and tonic DRN responses thus provide signals of consistent valence but over different timescales. The tonic changes in activity are consistent with previous data and hypotheses relating DRN activity to response suppression and impulse control. Phasic responses could contribute to this via online modulation of attention allocation through projections to sensory-processing regions.

Ventral striatal dopamine synthesis capacity predicts financial extravagance in Parkinson's disease

Impulse control disorders (ICDs), including disordered gambling, can occur in a significant number of patients with Parkinson’s disease (PD) receiving dopaminergic therapy. The neurobiology underlying susceptibility to such problems is unclear, but risk likely results from an interaction between dopaminergic medication and a pre-existing trait vulnerability. Impulse control and addictive disorders form part of a broader psychopathological spectrum of disorders, which share a common underlying genetic vulnerability, referred to as externalizing. The broad externalizing risk factor is a continuously varying trait reflecting vulnerability to various impulse control problems, manifested at the overt level by disinhibitory symptoms and at the personality level by antecedent traits such as impulsivity and novelty/sensation seeking. Trait “disinhibition” is thus a core endophenotype of ICDs, and a key target for neurobiological investigation. The ventral striatal dopamine system has been hypothesized to underlie individual variation in behavioral disinhibition. Here, we examined whether individual differences in ventral striatal dopamine synthesis capacity predicted individual variation in disinhibitory temperament traits in individuals with PD. Eighteen early-stage male PD patients underwent 6-[¹⁸F]Fluoro-l-DOPA (FDOPA) positron emission tomography scanning to measure striatal dopamine synthesis capacity, and completed a measure of disinhibited personality. Consistent with our predictions, we found that levels of ventral, but not dorsal, striatal dopamine synthesis capacity predicted disinhibited personality, particularly a propensity for financial extravagance. Our results are consistent with recent preclinical models of vulnerability to behavioral disinhibition and addiction proneness, and provide novel insights into the neurobiology of potential vulnerability to impulse control problems in PD and other disorders.

Instrumental vigour in punishment and reward

Recent notions about the vigour of responding in operant conditioning suggest that the long-run average rate of reward should control the alacrity of action in cases in which the actual cost of speed is balanced against the opportunity cost of sloth. The average reward rate is suggested as being reported by tonic activity in the dopamine system and thereby influencing all actions, including ones that do not themselves lead directly to the rewards. This idea is syntactically problematical for the case of punishment. Here, we broaden the scope of the original suggestion, providing a two-factor analysis of obviated punishment in a variety of operant circumstances. We also consider the effects of stochastically successful actions, which turn out to differ rather markedly between appetitive and aversive cases. Finally, we study how to fit these ideas into nascent treatments that extend concepts of opponency between dopamine and serotonin from valence to invigoration.

A subpopulation of serotonergic neurons that do not express the 5-HT1A autoreceptor

5-HT neurons are topographically organized in the hindbrain, and have been implicated in the etiology and treatment of psychiatric diseases such as depression and anxiety. Early studies suggested that the raphe 5-HT neurons were a homogeneous population showing similar electrical properties, and feedback inhibition mediated by 5-HT1A autoreceptors. We utilized histochemistry techniques in ePet1-eGFP and 5-HT1A-iCre/R26R mice to show that a subpopulation of 5-HT neurons do not express the somatodendritic 5-HT1A autoreceptor mRNA. In addition, we performed patch-clamp recordings followed by single-cell PCR in ePet1-eGFP mice. From 134 recorded 5-HT neurons located in the dorsal, lateral, and median raphe, we found lack of 5-HT1A mRNA expression in 22 cells, evenly distributed across raphe subfields. We compared the cellular characteristics of these neuronal types and found no difference in passive membrane properties and general excitability. However, when injected with large depolarizing current, 5-HT1A-negative neurons fired more action potentials, suggesting a lack of autoinhibitory action of local 5-HT release. Our results support the hypothesis that the 5-HT system is composed of subpopulations of serotonergic neurons with different capacity for adaptation.

Twenty-five lessons from computational neuromodulation

Neural processing faces three rather different, and perniciously tied, communication problems. First, computation is radically distributed, yet point-to-point interconnections are limited. Second, the bulk of these connections are semantically uniform, lacking differentiation at their targets that could tag particular sorts of information. Third, the brain's structure is relatively fixed, and yet different sorts of input, forms of processing, and rules for determining the output are appropriate under different, and possibly rapidly changing, conditions. Neuromodulators address these problems by their multifarious and broad distribution, by enjoying specialized receptor types in partially specific anatomical arrangements, and by their ability to mold the activity and sensitivity of neurons and the strength and plasticity of their synapses. Here, I offer a computationally focused review of algorithmic and implementational motifs associated with neuromodulators, using decision making in the face of uncertainty as a running example.

The influence of serotonin on fear learning

Learning of associations between aversive stimuli and predictive cues is the basis of Pavlovian fear conditioning and is driven by a mismatch between expectation and outcome. To investigate whether serotonin modulates the formation of such aversive cue-outcome associations, we used functional magnetic resonance imaging (fMRI) and dietary tryptophan depletion to reduce brain serotonin (5-HT) levels in healthy human subjects. In a Pavlovian fear conditioning paradigm, 5-HT depleted subjects compared to a non-depleted control group exhibited attenuated autonomic responses to cues indicating the upcoming of an aversive event. These results were closely paralleled by reduced aversive learning signals in the amygdala and the orbitofrontal cortex, two prominent structures of the neural fear circuit. In agreement with current theories of serotonin as a motivational opponent system to dopamine in fear learning, our data provide first empirical evidence for a role of serotonin in representing formally derived learning signals for aversive events.

Activation of dorsal raphe serotonin neurons is necessary for waiting for delayed rewards

The forebrain serotonergic system is a crucial component in the control of impulsive behaviors. We previously reported that the activity of serotonin neurons in the midbrain dorsal raphe nucleus increased when rats performed a task that required them to wait for delayed rewards. However, the causal relationship between serotonin neural activity and the tolerance for the delayed reward remained unclear. Here, we test whether the inhibition of serotonin neural activity by the local application of the 5-HT(1A) receptor agonist 8-hydroxy-2-(di-n-propylamino) tetralin in the dorsal raphe nucleus impairs rats' tolerance for delayed rewards. Rats performed a sequential food-water navigation task that required them to visit food and water sites alternately via a tone site to get rewards at both sites after delays. During the short (2 s) delayed reward condition, the inhibition of serotonin neural activity did not significantly influence the numbers of reward choice errors (nosepoke at an incorrect reward site following a conditioned reinforcer tone), reward wait errors (failure to wait for the delayed rewards), or total trials (sum of reward choice errors, reward wait errors, and acquired rewards). By contrast, during the long (7-11 s) delayed reward condition, the number of wait errors significantly increased while the numbers of total trials and choice errors did not significantly change. These results indicate that the activation of dorsal raphe serotonin neurons is necessary for waiting for long delayed rewards and suggest that elevated serotonin activity facilitates waiting behavior when there is the prospect of forthcoming rewards.

Similar improvement of reward and punishment learning by serotonin reuptake inhibitors in obsessive-compulsive disorder

BACKGROUND

The role of dopamine in reinforcement learning has been extensively studied, but the role of other major neuromodulators, particularly serotonin, remains poorly understood. An influential theory has suggested that dopamine and serotonin represent opponent systems respectively driving reward and punishment learning.

METHODS

To test this theory, we compared two groups of patients with obsessive-compulsive disorder, one unmedicated (n = 12) and one treated with serotonin reuptake inhibitors (SRI; n = 13). To avoid confounding basic reinforcement learning with strategic conscious reasoning, we used a subliminal conditioning task that involves subjects learning to associate masked cues with gambling outcomes to maximize their payoff. The same task was used in a previous study to demonstrate opposite effects of dopaminergic medication on reward and punishment learning.

RESULTS

Unmedicated obsessive-compulsive disorder patients exhibited an instrumental learning deficit that was fully alleviated under SRI treatment. Contrary to dopaminergic medication, SRIs similarly modulated reward and punishment learning.

CONCLUSIONS

Thus, departing from the opponency model, our results support a beneficial role of serotonin in instrumental learning that is independent of outcome valence.

Behavioural and pharmacological effects of fluvoxamine on decision-making in food patches and the inter-temporal choices of domestic chicks

Behavioural effects of fluvoxamine (FLV, selective serotonin reuptake inhibitor) were examined in 1-2 week old domestic chicks. Chicks were tested in an I-shaped maze equipped with a feeder (ON feeder) that served 1 or 2 grains of millet at gradually increasing intervals, so that a depleting food patch was mimicked. By leaving the feeder, the food delivery program was reset, and chicks gained food at short intervals only after a travel to a dummy feeder (OFF feeder) placed on the opposite side of the maze. Chicks quickly learned to actively shuttle between the ON and the OFF feeders. FLV (intra-peritoneal injection, 20 mg/kg BW) acutely caused chicks to stay longer at the gradually depleting ON feeder. Inter-temporal choices were also tested, whereby two coloured beads were simultaneously presented, each associated with a small/short-delay reward or a large/long-delay alternative. FLV suppressed the choice of the short-delay option. It is suggested that an enhanced level of serotonin (5-HT) makes chicks more tolerant of the delayed food item in both behavioural paradigms. Furthermore, the decision to leave a depleting patch cannot be equated to choosing the long-delay option of the choice paradigm. Furthermore, FLV suppressed work efforts (velocity and running distance) in uncued shuttle and number of distress calls. In vivo microdialysis experiments revealed that FLV enhanced the extracellular concentration of 5-HT as well as dopamine (DA) locally in the medial striatum/nucleus accumbens. Underlying neuromodulatory mechanisms of behavioural control are examined in relation to locomotion, behavioural tolerance and interval timing.

Dopamine, serotonin and impulsivity

Impulsive people have a strong urge to act without thinking. It is sometimes regarded as a positive trait but rash impulsiveness is also widely present in clinical disorders such as attention deficit hyperactivity disorder (ADHD), drug dependence, mania, and antisocial behaviour. Contemporary research has begun to make major inroads into unravelling the brain mechanisms underlying impulsive behaviour with a prominent focus on the limbic cortico-striatal systems. With this progress has come the understanding that impulsivity is a multi-faceted behavioural trait involving neurally and psychologically diverse elements. We discuss the significance of this heterogeneity for clinical disorders expressing impulsive behaviour and the pivotal contribution made by the brain dopamine and serotonin systems in the aetiology and treatment of behavioural syndromes expressing impulsive symptoms.

Serotonin selectively modulates reward value in human decision-making

Establishing a function for the neuromodulator serotonin in human decision-making has proved remarkably difficult because if its complex role in reward and punishment processing. In a novel choice task where actions led concurrently and independently to the stochastic delivery of both money and pain, we studied the impact of decreased brain serotonin induced by acute dietary tryptophan depletion. Depletion selectively impaired both behavioral and neural representations of reward outcome value, and hence the effective exchange rate by which rewards and punishments were compared. This effect was computationally and anatomically distinct from a separate effect on increasing outcome-independent choice perseveration. Our results provide evidence for a surprising role for serotonin in reward processing, while illustrating its complex and multifarious effects.

Genetic depletion of brain 5HT reveals a common molecular pathway mediating compulsivity and impulsivity

Neuropsychiatric disorders characterized by behavioral disinhibition, including disorders of compulsivity (e.g. obsessive-compulsive disorder; OCD) and impulse-control (e.g. impulsive aggression), are severe, highly prevalent and chronically disabling. Treatment options for these diseases are extremely limited. The pathophysiological bases of disorders of behavioral disinhibition are poorly understood but it has been suggested that serotonin dysfunction may play a role. Mice lacking the gene encoding brain tryptophan hydroxylase 2 (Tph2-/-), the initial and rate-limiting enzyme in the synthesis of serotonin, were tested in numerous behavioral assays that are well known for their utility in modeling human neuropsychiatric diseases. Mice lacking Tph2 (and brain 5HT) show intense compulsive and impulsive behaviors to include extreme aggression. The impulsivity is motor in form and not cognitive because Tph2-/- mice show normal acquisition and reversal learning on a spatial learning task. Restoration of 5HT levels by treatment of Tph2-/- mice with its immediate precursor 5-hydroxytryptophan attenuated compulsive and impulsive-aggressive behaviors. Surprisingly, in Tph2-/- mice, the lack of 5HT was not associated with anxiety-like behaviors. The results indicate that 5HT mediates behavioral disinhibition in the mammalian brain independent of anxiogenesis.

Investigation of the neuroanatomical substrates of reward seeking following protracted abstinence in mice

Persistent vulnerability to relapse represents a major challenge in the treatment of drug addiction. The brain circuitry that underlies relapse-like behaviour can be investigated using animal models of drug seeking. As yet there have been no comprehensive brain mapping studies that have specifically examined the neuroanatomical substrates of cue-induced opiate seeking following abstinence in a mouse operant paradigm. The aim of this study was to compare the brain regions involved in sucrose vs. morphine seeking following protracted abstinence in mice. Male CD1 mice were trained to respond for either sucrose (10% w/v) or intravenous morphine (0.1 mg kg(-1) per infusion) in an operant paradigm in the presence of a discrete cue. Once stable responding was established, mice were subjected to abstinence in their home cages for 3 weeks and then perfused for tissue collection, or returned to the operant chambers to assess cue-induced reward seeking before being perfused for tissue collection. Brain tissue was processed for Fos immunohistochemistry and Fos expression was quantified in a range of brain nuclei. We identified unique patterns of neuronal activation for sucrose and morphine seeking mice as well as some overlap. Structures activated in both ‘relapse' groups included the anterior cingulate and orbitofrontal cortex, nucleus accumbens shell, bed nucleus of the stria terminalis, substantia nigra pars compacta, ventral tegmental area, hippocampus, periaqueductal grey, locus coeruleus and lateral habenula. Structures that were more activated in morphine seeking mice included the nucleus accumbens core, basolateral amygdala, substantia nigra pars reticulata, and the central nucleus of the amygdala. The dorsal raphe was the only structure examined that was specifically activated in sucrose seeking mice. Overall our findings support a cortico-striatal limbic circuit driving opiate seeking, and we have identified some additional circuitry potentially relevant to reward seeking following abstinence.

Properties of I(A) in a neuron of the dorsal raphe nucleus

Voltage clamp data were analyzed in order to characterize the properties of the fast potassium transient current I(A) for a presumed serotonergic neuron of the rat dorsal raphe nucleus (DRN). We obtain maximal conductance, time constants of activation and inactivation, and the steady state activation and inactivation functions m(∞) and h(∞), as Boltzmann curves, defined by half-activation potentials and slope factors. I(A) is estimated as g¯(V-V(rev))m(4)h, with g¯=20.5nS. For activation, the half-activation potential is V(a)=-52.5mV with slope factor k(a)=16.5mV, whereas for inactivation the corresponding quantities are -91.5mV and -9.3mV. We discuss the results in terms of the corresponding properties of I(A) in other cell types and their possible relevance to pacemaking activity in cells of the DRN. Methods of identification of serotonergic DRN neurons and the nature of the K(v) channels underlying the A-type current are also discussed.

The role of serotonin in the regulation of patience and impulsivity

Classic theories suggest that central serotonergic neurons are involved in the behavioral inhibition that is associated with the prediction of negative rewards or punishment. Failed behavioral inhibition can cause impulsive behaviors. However, the behavioral inhibition that results from predicting punishment is not sufficient to explain some forms of impulsive behavior. In this article, we propose that the forebrain serotonergic system is involved in “waiting to avoid punishment” for future punishments and “waiting to obtain reward” for future rewards. Recently, we have found that serotonergic neurons increase their tonic firing rate when rats await food and water rewards and conditioned reinforcer tones. The rate of tonic firing during the delay period was significantly higher when rats were waiting for rewards than for tones, and rats were unable to wait as long for tones as for rewards. These results suggest that increased serotonergic neuronal firing facilitates waiting behavior when there is the prospect of a forthcoming reward and that serotonergic activation contributes to the patience that allows rats to wait longer. We propose a working hypothesis to explain how the serotonergic system regulates patience while waiting for future rewards.

Impact of size and delay on neural activity in the rat limbic corticostriatal system

A number of factors influence an animal’s economic decisions. Two most commonly studied are the magnitude of and delay to reward. To investigate how these factors are represented in the firing rates of single neurons, we devised a behavioral task that independently manipulated the expected delay to and size of reward. Rats perceived the differently delayed and sized rewards as having different values and were more motivated under short delay and big-reward conditions than under long delay and small reward conditions as measured by percent choice, accuracy, and reaction time. Since the creation of this task, we have recorded from several different brain areas including, orbitofrontal cortex, striatum, amygdala, substantia nigra pars reticulata, and midbrain dopamine neurons. Here, we review and compare those data with a substantial focus on those areas that have been shown to be critical for performance on classic time discounting procedures and provide a potential mechanism by which they might interact when animals are deciding between differently delayed rewards. We found that most brain areas in the cortico-limbic circuit encode both the magnitude and delay to reward delivery in one form or another, but only a few encode them together at the single neuron level.

A neural correlate of predicted and actual reward-value information in monkey pedunculopontine tegmental and dorsal raphe nucleus during saccade tasks

Dopamine, acetylcholine, and serotonin, the main modulators of the central nervous system, have been proposed to play important roles in the execution of movement, control of several forms of attentional behavior, and reinforcement learning. While the response pattern of midbrain dopaminergic neurons and its specific role in reinforcement learning have been revealed, the role of the other neuromodulators remains rather elusive. Here, we review our recent studies using extracellular recording from neurons in the pedunculopontine tegmental nucleus, where many cholinergic neurons exist, and the dorsal raphe nucleus, where many serotonergic neurons exist, while monkeys performed eye movement tasks to obtain different reward values. The firing patterns of these neurons are often tonic throughout the task period, while dopaminergic neurons exhibited a phasic activity pattern to the task event. The different modulation patterns, together with the activity of dopaminergic neurons, reveal dynamic information processing between these different neuromodulator systems.

Increased expression of 5-HT₆ receptors in dorsolateral striatum decreases habitual lever pressing, but does not affect learning acquisition of simple operant tasks in rats

Serotonin-6 (5-HT(6)) receptors are densely expressed in the dorsolateral striatum (DLS), a brain region linked to habits. Medications acting on the serotonergic system, including 5-HT(6) receptors, can diminish habitual and repetitive behaviors associated with clinical syndromes such as obsessive-compulsive disorder, and may have implications for addiction as well. To examine the role of 5-HT(6) receptors in the acquisition and persistence of habitual behavior, we manipulated 5-HT(6) receptor expression in the DLS with herpes simplex virus vectors in combination with different behavioral procedures; control rats received a vector expressing enhanced green fluorescent protein. In one set of experiments, rats were tested under conditions that favor the acquisition of either discrete action-outcome responding or repetitive responding; increased 5-HT(6) receptor expression in DLS did not alter learning in either paradigm. In the next experiment, rats were over-trained on fixed- then variable-interval schedules, resulting in an escalation of lever pressing over sessions far in excess of that necessary to receive sucrose pellets. After training, rats received viral vector infusion into the DLS. Subsequently, half of each group underwent an omission contingency training session in which they received reinforcement for refraining from pressing the lever, while the other half served as yoked controls. A probe session under extinction conditions was performed the following day. Only rats that received both the 5-HT(6) vector and omission contingency training showed reduced lever pressing during the probe session. These results suggest that increasing 5-HT(6) receptor signaling in the DLS facilitates behavioral flexibility in the face of changing contingencies.

Allostasis: a model of predictive regulation

The premise of the standard regulatory model, "homeostasis", is flawed: the goal of regulation is not to preserve constancy of the internal milieu. Rather, it is to continually adjust the milieu to promote survival and reproduction. Regulatory mechanisms need to be efficient, but homeostasis (error-correction by feedback) is inherently inefficient. Thus, although feedbacks are certainly ubiquitous, they could not possibly serve as the primary regulatory mechanism. A newer model, "allostasis", proposes that efficient regulation requires anticipating needs and preparing to satisfy them before they arise. The advantages: (i) errors are reduced in magnitude and frequency; (ii) response capacities of different components are matched -- to prevent bottlenecks and reduce safety factors; (iii) resources are shared between systems to minimize reserve capacities; (iv) errors are remembered and used to reduce future errors. This regulatory strategy requires a dedicated organ, the brain. The brain tracks multitudinous variables and integrates their values with prior knowledge to predict needs and set priorities. The brain coordinates effectors to mobilize resources from modest bodily stores and enforces a system of flexible trade-offs: from each organ according to its ability, to each organ according to its need. The brain also helps regulate the internal milieu by governing anticipatory behavior. Thus, an animal conserves energy by moving to a warmer place - before it cools, and it conserves salt and water by moving to a cooler one before it sweats. The behavioral strategy requires continuously updating a set of specific "shopping lists" that document the growing need for each key component (warmth, food, salt, water). These appetites funnel into a common pathway that employs a "stick" to drive the organism toward filling the need, plus a "carrot" to relax the organism when the need is satisfied. The stick corresponds broadly to the sense of anxiety, and the carrot broadly to the sense of pleasure. This design constrains anxieties to be non-adapting and pleasures to be brief -- fast-adapting -- to make way for the next anxiety. The stick/carrot mechanisms evolved early and expanded so that in humans they govern higher level learning and social organization. Correspondingly, the "funnel" widened to allow innumerable activities and experiences to each provide non-adapting anxieties and brief pleasures, their reward values depending partly on the effort expended. But modern life narrows the variety of small pleasures and reduces effort, thereby reducing their reward value and requiring larger portions for equivalent satisfaction - a cycle that generates addictive behaviors. Homeostasis and allostasis locate pathology at different levels. Homeostasis identifies proximate causes; for example, it attributes essential hypertension to excess salt water in too small a vascular reservoir. Thus it directs pharmacotherapy toward reducing salt and water, expanding the reservoir, and blocking feedbacks that would counteract these measures. Allostasis attributes essential hypertension to the brain. Chronically anticipating a need for higher pressure, the brain mobilizes all the low level mechanisms in concert: kidney to retain salt and water, vascular system to tighten, and salt appetite to rise. Correspondingly, allostasis would direct therapy toward higher levels - to reduce demand and increase sense of control -- so that the brain can down-shift its prediction and relax all the low-level mechanisms in concert. For disorders of addiction homeostasis pursues pharmacological treatments: drugs to treat drug addiction, obesity, and other compulsive behaviors. Allostasis suggests broader approaches - such as re-expanding the range of possible pleasures and providing opportunities to expend effort in their pursuit.

Multiple representations and algorithms for reinforcement learning in the cortico-basal ganglia circuit

Accumulating evidence shows that the neural network of the cerebral cortex and the basal ganglia is critically involved in reinforcement learning. Recent studies found functional heterogeneity within the cortico-basal ganglia circuit, especially in its ventromedial to dorsolateral axis. Here we review computational issues in reinforcement learning and propose a working hypothesis on how multiple reinforcement learning algorithms are implemented in the cortico-basal ganglia circuit using different representations of states, values, and actions.