Activation of dorsal raphe serotonin neurons underlies waiting for delayed rewards

Fatigue modulates dopamine availability and promotes flexible choice reversals during decision making

During decisions, animals balance goal achievement and effort management. Despite physical exercise and fatigue significantly affecting the levels of effort that an animal exerts to obtain a reward, their role in effort-based choice and the underlying neurochemistry are incompletely known. In particular, it is unclear whether fatigue influences decision (cost-benefit) strategies flexibly or only post-decision action execution and learning. To answer this question, we trained mice on a T-maze task in which they chose between a high-cost, high-reward arm (HR), which included a barrier, and a low-cost, low-reward arm (LR), with no barrier. The animals were parametrically fatigued immediately before the behavioural tasks by running on a treadmill. We report a sharp choice reversal, from the HR to LR arm, at 80% of their peak workload (PW), which was temporary and specific, as the mice returned to choose the HC when the animals were successively tested at 60% PW or in a two-barrier task. These rapid reversals are signatures of flexible choice. We also observed increased subcortical dopamine levels in fatigued mice: a marker of individual bias to use model-based control in humans. Our results indicate that fatigue levels can be incorporated in flexible cost-benefits computations that improve foraging efficiency.

Cerebellar granule cells encode the expectation of reward

The human brain contains ~60 billion cerebellar granule cells¹, which outnumber all other neurons combined. Classical theories posit that a large, diverse population of granule cells allows for highly detailed representations of sensorimotor context, enabling downstream Purkinje cells to sense fine contextual changes^2–6. Although evidence suggests a role for cerebellum in cognition^7–10, granule cells are known to encode only sensory^11–13 and motor¹⁴ context. Using two-photon calcium imaging in behaving mice, here we show that granule cells convey information about the expectation of reward. Mice initiated voluntary forelimb movements for delayed water reward. Some granule cells responded preferentially to reward or reward omission, whereas others selectively encoded reward anticipation. Reward responses were not restricted to forelimb movement, as a Pavlovian task evoked similar responses. Compared to predictable rewards, unexpected rewards elicited markedly different granule cell activity despite identical stimuli and licking responses. In both tasks, reward signals were widespread throughout multiple cerebellar lobules. Tracking the same granule cells over several days of learning revealed that cells with reward-anticipating responses emerged from those that responded at the start of learning to reward delivery, whereas reward omission responses grew stronger as learning progressed. The discovery of predictive, non-sensorimotor encoding in granule cells is a major departure from current understanding of these neurons and dramatically enriches contextual information available to postsynaptic Purkinje cells, with important implications for cognitive processing in the cerebellum.

Transient inhibition and long-term facilitation of locomotion by phasic optogenetic activation of serotonin neurons

Serotonin (5-HT) is associated with mood and motivation but the function of endogenous 5-HT remains controversial. Here, we studied the impact of phasic optogenetic activation of 5-HT neurons in mice over time scales from seconds to weeks. We found that activating dorsal raphe nucleus (DRN) 5-HT neurons induced a strong suppression of spontaneous locomotor behavior in the open field with rapid kinetics (onset ≤1 s). Inhibition of locomotion was independent of measures of anxiety or motor impairment and could be overcome by strong motivational drive. Repetitive place-contingent pairing of activation caused neither place preference nor aversion. However, repeated 15 min daily stimulation caused a persistent increase in spontaneous locomotion to emerge over three weeks. These results show that 5-HT transients have strong and opposing short and long-term effects on motor behavior that appear to arise from effects on the underlying factors that motivate actions.

DOI:http://dx.doi.org/10.7554/eLife.20975.001

The brain controls sleep, movement and the other behaviors that an animal needs to survive. A chemical called serotonin plays an important role in controlling these behaviors as it regulates the activity of nerve cells (known as neurons) throughout the brain. Serotonin is produced by a specific group of neurons found in an area at the base of the brain called the raphe nuclei. From there, serotonin is released into other parts of the brain to influence different behaviors. Although drugs that target serotonin are widely used as antidepressants, how this chemical signal acts in the brain remains a mystery. This is due, in part, to it being technically challenging to carry out experiments on the serotonin-producing neurons.

A technique called optogenetics uses light to selectively activate or inhibit individual cells in live animals. Here, Correia, Lottem et al. use optogenetics to activate serotonin-producing neurons in the dorsal raphe nucleus of mice. The experiments show that triggering serotonin production for a few seconds causes the mice to move around more slowly as they explore their surroundings. This short-term release of serotonin only slows the mice down if they are not already occupied with other activities, such as finding water or balancing on a moving object. These experiments suggest that serotonin decreases an individual’s motivation to move but that this can be overcome by sufficiently powerful goals. In contrast, repeatedly activating the serotonin neurons over a period of several weeks led to long-term changes of the opposite kind – the mice begin to move around more quickly.

The findings of Correia, Lottem et al. have possible implications for the use of drugs that target serotonin to treat mental disorders as it suggests important links between serotonin, movement, and the ability of the brain to change how it responds to certain situations. The next steps will be to investigate how the two different effects of serotonin are connected, which areas in the brain are involved and how best to apply these findings to clinical studies.

DOI:http://dx.doi.org/10.7554/eLife.20975.002

An active inference theory of allostasis and interoception in depression

In this paper, we integrate recent theoretical and empirical developments in predictive coding and active inference accounts of interoception (including the Embodied Predictive Interoception Coding model) with working hypotheses from the theory of constructed emotion to propose a biologically plausible unified theory of the mind that places metabolism and energy regulation (i.e. allostasis), as well as the sensory consequences of that regulation (i.e. interoception), at its core. We then consider the implications of this approach for understanding depression. We speculate that depression is a disorder of allostasis, whose myriad symptoms result from a 'locked in' brain that is relatively insensitive to its sensory context. We conclude with a brief discussion of the ways our approach might reveal new insights for the treatment of depression.This article is part of the themed issue 'Interoception beyond homeostasis: affect, cognition and mental health'.

Input-specific contributions to valence processing in the amygdala

Reward and punishment are often thought of as opposing processes: rewards and the environmental cues that predict them elicit approach and consummatory behaviors, while punishments drive aversion and avoidance behaviors. This framework suggests that there may be segregated brain circuits for these valenced behaviors. The basolateral amygdala (BLA) is one brain region that contributes to both types of motivated behavior. Individual neurons in the BLA can favor positive over negative valence, or vice versa, but these neurons are intermingled, showing no anatomical segregation. The amygdala receives inputs from many brain areas and current theories posit that encoding of positive versus negative valence by BLA neurons is determined by the wiring of each neuron. Specifically, many projections from other brain areas that respond to positive and negative valence stimuli and predictive cues project strongly to the BLA and likely contribute to valence processing within the BLA. Here we review three of these areas, the basal forebrain, the dorsal raphe nucleus and the ventral tegmental area, and discuss how these may promote encoding of positive and negative valence within the BLA.

Blocking serotonin but not dopamine reuptake alters neural processing during perceptual decision making

Dopamine and serotonin have opponent interactions on aspects of impulsivity. Therefore we wanted to test the hypothesis that dopamine and serotonin would have opposing effects on speed-accuracy trade offs in a perceptual decision making task. Unlike other behavioral measures of impulsivity, perceptual decision making allows us to determine whether decreasing premature responses, often interpreted as decreased impulsivity, corresponds to increased behavioral performance. We administered GBR-12909 (a dopamine transporter blocker), escitalopram (a serotonin transporter blocker), or saline in separate sessions to 3 rhesus macaques. We found that animals had slower reaction times (RTs) on escitalopram than on GBR-12909 or saline. However, they were also least accurate on escitalopram. Animals were faster, although nonsignificantly, on GBR than saline and had equivalent accuracy. Administration of GBR-12909 did cause animals to be faster in error trials than correct trials. Therefore, from the point of view of RTs the animals were less impulsive on escitalopram. However, the decreased accuracy of the monkeys shows that they were not able to make use of their slower response times to make more accurate decisions. Therefore, impulsivity was reduced on escitalopram, but at the expense of a slower information-processing rate in the perceptual inference task. (PsycINFO Database Record

Neural activity of orbitofrontal cortex contributes to control of waiting

The willingness to wait for delayed reward and information is of fundamental importance for deliberative behaviors. The orbitofrontal cortex (OFC) is thought to be a core component of the neural circuitry underlying the capacity to control waiting. However, the neural correlates of active waiting and the causal role of the OFC in the control of waiting still remain largely unknown. Here, we trained rats to perform a waiting task (waiting for a pseudorandom time to obtain the water reward), and recorded neuronal ensembles in the OFC throughout the task. We observed that subset OFC neurons exhibited ramping activities throughout the waiting process. Receiver operating characteristic analysis showed that neural activities during the waiting period even predicted the trial outcomes (patient vs. impatient) on a trial-by-trial basis. Furthermore, optogenetic activation of the OFC during the waiting period improved the waiting performance, but did not influence rats' movement to obtain the reward. Taken together, these findings reveal that the neural activity in the OFC contributes to the control of waiting.

The neural underpinnings of cognitive flexibility and their disruption in psychotic illness

Schizophrenia (SZ) has long been associated with a variety of cognitive deficits, including reduced cognitive flexibility. More recent findings, however, point to tremendous inter-individual variability among patients on measures of cognitive flexibility/set-shifting. With an eye toward shedding light on potential sources of variability in set-shifting abilities among SZ patients, I examine the neural substrates of underlying probabilistic reversal learning (PRL) - a paradigmatic measure of cognitive flexibility - as well as neuromodulatory influences upon these systems. Finally, I report on behavioral and neuroimaging studies of PRL in SZ patients, discussing the potentially influences of illness profile and antipsychotic medications on cognitive flexibility in SZ.

Serotonin neurons in the dorsal raphe nucleus encode reward signals

The dorsal raphe nucleus (DRN) is involved in organizing reward-related behaviours; however, it remains unclear how genetically defined neurons in the DRN of a freely behaving animal respond to various natural rewards. Here we addressed this question using fibre photometry and single-unit recording from serotonin (5-HT) neurons and GABA neurons in the DRN of behaving mice. Rewards including sucrose, food, sex and social interaction rapidly activate 5-HT neurons, but aversive stimuli including quinine and footshock do not. Both expected and unexpected rewards activate 5-HT neurons. After mice learn to wait for sucrose delivery, most 5-HT neurons fire tonically during waiting and then phasically on reward acquisition. Finally, GABA neurons are activated by aversive stimuli but inhibited when mice seek rewards. Thus, DRN 5-HT neurons positively encode a wide range of reward signals during anticipatory and consummatory phases of reward responses. Moreover, GABA neurons play a complementary role in reward processing.

Multiscale single-cell analysis reveals unique phenotypes of raphe 5-HT neurons projecting to the forebrain

Serotonergic neurons of the raphe nuclei exhibit anatomical, neurochemical and elecrophysiological heterogeneity that likely underpins their specific role in multiple behaviors. However, the precise organization of serotonin (5-HT) neurons to orchestrate 5-HT release patterns throughout the brain is not well understood. We compared the electrophysiological and neurochemical properties of dorsal and median raphe 5-HT neurons projecting to the medial prefrontal cortex (mPFC), amygdala (BLA) and dorsal hippocampus (dHP), combining retrograde tract tracing with brain slice electrophysiology and single-cell RT-PCR in Pet1-EGFP mice. Our results show that 5-HT neurons projecting to the dHP and the mPFC and the BLA form largely non-overlapping populations and that BLA-projecting neurons have characteristic excitability and membrane properties. In addition, using an unbiased clustering method that correlates anatomical, molecular and electrophysiological phenotypes, we find that 5-HT neurons with projections to the mPFC and the dHP segregate from those projecting to the BLA. Single-cell gene profiling showed a restricted expression of the peptide galanin in the population of 5-HT neurons projecting to the mPFC. Finally, cluster analysis allowed identifying an atypical subtype of 5-HT neuron with low excitability, long firing delays and preferential expression of the vesicular glutamate transporter type 3. Overall, these findings allow to define correlated anatomical and physiological identities of serotonin raphe neurons that help understanding how discrete raphe cells subpopulations account for the heterogeneous activities of the midbrain serotonergic system.

Cellular resilience: 5-HT neurons in Tph2(-/-) mice retain normal firing behavior despite the lack of brain 5-HT

Considerable evidence links dysfunction of serotonin (5-hydroxytryptamine, 5-HT) transmission to neurodevelopmental and psychiatric disorders characterized by compromised "social" cognition and emotion regulation. It is well established that the brain 5-HT system is under autoregulatory control by its principal transmitter 5-HT via its effects on activity and expression of 5-HT system-related proteins. To examine whether 5-HT itself also has a crucial role in the acquisition and maintenance of characteristic rhythmic firing of 5-HT neurons, we compared their intrinsic electrophysiological properties in mice lacking brain 5-HT, i.e. tryptophan hydroxylase-2 null mice (Tph2(-/-)) and their littermates, Tph2(+/-) and Tph2(+/+), by using whole-cell patch-clamp recordings in a brainstem slice preparation and single unit recording in anesthetized animals. We report that the active properties of dorsal raphe nucleus (DRN) 5-HT neurons in vivo (firing rate magnitude and variability; the presence of spike doublets) and in vitro (firing in response to depolarizing current pulses; action potential shape) as well as the resting membrane potential remained essentially unchanged across Tph2 genotypes. However, there were subtle differences in subthreshold properties, most notably, an approximately 25% higher input conductance in Tph2(-/-) mice compared with Tph2(+/-) and Tph2(+/+) littermates (p<0.0001). This difference may at least in part be a consequence of slightly bigger size of the DRN 5-HT neurons in Tph2(-/-) mice (approximately 10%, p<0.0001). Taken together, these findings show that 5-HT neurons acquire and maintain their signature firing properties independently of the presence of their principal neurotransmitter 5-HT, displaying an unexpected functional resilience to complete brain 5-HT deficiency.

Serotonin: waiting but not rewarding

A new study in mice demonstrates that stimulating dorsal raphe serotonin boosts patient waiting but stimulation itself is not rewarding. Dorsal raphe serotonin's unique contribution provides a neural locus for impulsivity and related failures of patience.

Reward processing by the dorsal raphe nucleus: 5-HT and beyond

The dorsal raphe nucleus (DRN) represents one of the most sensitive reward sites in the brain. However, the exact relationship between DRN neuronal activity and reward signaling has been elusive. In this review, we will summarize anatomical, pharmacological, optogenetics, and electrophysiological studies on the functions and circuit mechanisms of DRN neurons in reward processing. The DRN is commonly associated with serotonin (5-hydroxytryptamine; 5-HT), but this nucleus also contains neurons of the neurotransmitter phenotypes of glutamate, GABA and dopamine. Pharmacological studies indicate that 5-HT might be involved in modulating reward- or punishment-related behaviors. Recent optogenetic stimulations demonstrate that transient activation of DRN neurons produces strong reinforcement signals that are carried out primarily by glutamate. Moreover, activation of DRN 5-HT neurons enhances reward waiting. Electrophysiological recordings reveal that the activity of DRN neurons exhibits diverse behavioral correlates in reward-related tasks. Studies so far thus demonstrate the strong power of DRN neurons in reward signaling and at the same time invite additional efforts to dissect the roles and mechanisms of different DRN neuron types in various processes of reward-related behaviors.

5-HT1A Receptor-Mediated Autoinhibition and the Control of Serotonergic Cell Firing

The idea that serotonergic synaptic transmission plays an essential role in the control of mood and the pharmacotherapy of anxiety and depression is one of the cornerstones of modern biological psychiatry. As a result, there is intense interest in understanding the mechanisms controlling the activity of serotonin-synthesizing (serotonergic) neurons. One of the oldest and most durable ideas emerging from this work is that serotonergic neurons are capable of autonomously regulating their own basal firing rate. Serotonergic neurons express on their surface 5-HT1A receptors (autoreceptors) that, when activated, induce the opening of potassium channels that hyperpolarize and thereby inhibit cell firing. Activity-dependent release of serotonin within serotonergic nuclei is thought to activate these autoreceptors, thus completing an autoinhibitory feedback loop. This concept, which was originally proposed in the 1970s, has proven to be enormously fruitful and has guided the interpretation of a broad range of clinical and preclinical work. Yet, remarkably, electrophysiological studies seeking to directly demonstrate this phenomenon, especially in in vitro brain slices, have produced mixed results. Here, we critically review this work with a focus on electrophysiological studies, which directly assess neuronal activity. We also highlight recent work suggesting that 5-HT1A receptor-mediated autoinhibition may play other roles in the control of firing besides acting as a feedback regulator for the pacemaker-like firing rate of serotonergic neurons.

Appetitive and aversive information coding in the primate dorsal raphé nucleus

Serotonin is known to play a key role in the regulation of emotional behavior. There have been conflicting hypotheses about whether the central serotonergic system is involved in positive or negative emotional information processing. To reveal whether and how such opposing information processing can be achieved by single neurons in the dorsal raphé nucleus (DRN), the major source of serotonin in the forebrain, we recorded the activity of DRN neurons while monkeys were conditioned in a Pavlovian procedure with two distinct contexts: an appetitive block where a reward was available; and an aversive one where an airpuff was delivered. We found that single DRN neurons were involved in several aspects of both appetitive and aversive information processing. First, more than half of the recorded DRN neurons discriminated between appetitive and aversive contexts by tonic changes in their activity. In the appetitive context, they then kept track of the expected reward value indicated by the conditioned stimuli. Some of them also encoded an error between the obtained and expected values. In the aversive context, the same neurons maintained tonic modulation in their activity throughout the block. However, modulation of their responses to aversive task events depending on airpuff probability was less common. Together, these results indicate that single DRN neurons encode both appetitive and aversive information, but over differing time scales: relatively shorter for appetitive, and longer for aversive. Such temporally distinct processes of value coding in the DRN may provide the neural basis of emotional information processing in different contexts.

Neuroscience: waiting for serotonin

Serotonin dysfunction is implicated in many neuropsychiatric disorders yet the precise behavioral functions of this neuromodulator are not well understood. A new study employs optogenetic methods to activate serotonin neurons during an effort-demanding waiting behavior and demonstrates that serotonin release increases patience, the capacity for self-control.

Ramping single unit activity in the medial prefrontal cortex and ventral striatum reflects the onset of waiting but not imminent impulsive actions

The medial prefrontal cortex (mPFC) and ventral striatum (VS), including the nucleus accumbens, are key forebrain regions involved in regulating behaviour for future rewards. Dysfunction of these regions can result in impulsivity, characterized by actions that are mistimed and executed without due consideration of their consequences. Here we recorded the activity of single neurons in the mPFC and VS of 16 rats during performance on a five-choice serial reaction time task of sustained visual attention and impulsivity. Impulsive responses were assessed by the number of premature responses made before target stimuli were presented. We found that the majority of cells signalled trial outcome after an action was made (both rewarded and unrewarded). Positive and negative ramping activity was a feature of population activity in the mPFC and VS (49.5 and 50.4% of cells, respectively). This delay-related activity increased at the same rate and reached the same maximum (or minimum) for trials terminated by either correct or premature responses. However, on premature trials, the ramping activity started earlier and coincided with shorter latencies to begin waiting. For all trial types the pattern of ramping activity was unchanged when the pre-stimulus delay period was made variable. Thus, premature responses may result from a failure in the timing of the initiation of a waiting process, combined with a reduced reliance on external sensory cues, rather than a primary failure in delay activity. Our findings further show that the neural locus of this aberrant timing signal may emanate from structures outside the mPFC and VS.

Prospective coding of dorsal raphe reward signals by the orbitofrontal cortex

The orbitofrontal cortex (OFC) is important for the cognitive processes of learning and decision making. Previous recordings have revealed that OFC neurons encode predictions of reward outcomes. The OFC is interconnected with the dorsal raphe nucleus (DRN), which is a major serotonin (5-HT) center of the brain. Recent studies have provided increasing evidence that the DRN encodes reward signals. However, it remains unclear how the activity of DRN neurons affects the prospective reward coding of OFC neurons. By combining single-unit recordings from the OFC and optogenetic activation of the DRN in behaving mice, we found that DRN stimulation is sufficient to organize and modulate the anticipatory responses of OFC neurons. During pavlovian conditioning tasks for mice, odorant cues were associated with the delayed delivery of natural rewards of sucrose solution or DRN stimulation. After training, OFC neurons exhibited prospective responses to the sucrose solution. More importantly, the coupling of an odorant with delayed DRN stimulation resulted in tonic excitation or inhibition of OFC neurons during the delay period. The intensity of the prospective responses was affected by the frequency and duration of DRN stimulation. Additionally, DRN stimulation bidirectionally modulated the prospective responses to natural rewards. These experiments indicate that signals from the DRN are incorporated into the brain reward system to shape the cortical prospective coding of rewards.

Serotonergic neurons signal reward and punishment on multiple timescales

Serotonin's function in the brain is unclear. One challenge in testing the numerous hypotheses about serotonin's function has been observing the activity of identified serotonergic neurons in animals engaged in behavioral tasks. We recorded the activity of dorsal raphe neurons while mice experienced a task in which rewards and punishments varied across blocks of trials. We ‘tagged’ serotonergic neurons with the light-sensitive protein channelrhodopsin-2 and identified them based on their responses to light. We found three main features of serotonergic neuron activity: (1) a large fraction of serotonergic neurons modulated their tonic firing rates over the course of minutes during reward vs punishment blocks; (2) most were phasically excited by punishments; and (3) a subset was phasically excited by reward-predicting cues. By contrast, dopaminergic neurons did not show firing rate changes across blocks of trials. These results suggest that serotonergic neurons signal information about reward and punishment on multiple timescales.

DOI:http://dx.doi.org/10.7554/eLife.06346.001

Rewards and punishments can both encourage animals to change their immediate behavior and influence their mood over a longer term, particularly when given repeatedly. A region of the brain that increases its activity in response to rewards and punishments also contains many neurons that communicate with each other by releasing a chemical called serotonin. This chemical is commonly thought to produce feelings of happiness; however, it remains unclear exactly how these particular ‘serotonergic’ neurons help to process rewards and punishments.

The ideal way to work out the role that a type of neuron plays in a behavior is to measure its electrical activity as the behavior is being performed. However, it is difficult to distinguish the activity of serotonergic neurons from the activity of the non-serotonergic neurons around them. To overcome this problem, Cohen et al. used viruses to force serotonergic neurons to make a type of ion channel that produces electrical currents in response to light. Shining light on these neurons via optical fibers and then measuring the neurons' responses helped to develop criteria that can identify which responses are generated by the serotonergic neurons.

Cohen et al. then recorded the activity of serotonergic neurons in thirsty mice as they experienced a series of rewards (for example, a drop of water) or punishments (such as a puff of air to the eye). Each reward or punishment was preceded by a distinct odor, so that the mice learned to anticipate what was coming.

These experiments revealed that serotonergic neurons respond to rewards and punishments by changing two aspects of their electrical activity: by producing short bursts of high activity, and by altering their baseline activity. Some of the serotonergic neurons fired rapidly in response to punishments, but not rewards; others fired rapidly when the mice detected a scent that meant that a reward was about to be given. The average level of reward or punishment the mice received also affected the baseline activity of many of the serotonergic neurons; this effect lasted for several minutes.

Overall, Cohen et al. suggest that serotonergic neurons can affect how mice respond to rewards or punishments in both the short and long term. Future experiments should aim to understand the diversity of the signals that Cohen et al. observed, and to determine how these signals are used to drive behavior. Ultimately, understanding how neural circuits made up of different types of cells work may aid in understanding the neural basis of behavior.

DOI:http://dx.doi.org/10.7554/eLife.06346.002

Translatable and Back-Translatable Measurement of Impulsivity and Compulsivity: Convergent and Divergent Processes

Impulsivity and compulsivity have emerged as important dimensional constructs that challenge traditional psychiatric classification systems. Both are present in normal healthy populations where the need to act quickly and repeatedly without hesitation can be highly advantageous. However, when excessively expressed, impulsive and compulsive behavior can lead to adverse consequences and spectrum disorders exemplified by attention-deficit/hyperactivity disorder (ADHD), obsessive compulsive disorder (OCD), autism, and drug addiction. Impulsive individuals have difficulty in deferring gratification and are inclined to 'jump the gun' and respond prematurely before sufficient information is gathered. Compulsivity involves repetitive behavior often motivated by the need to reduce or prevent anxiety, thus leading to the maladaptive perseveration of behavior. Defined in this way, impulsivity and compulsivity could be viewed as separate entities or 'traits' but overwhelming evidence indicates that both may be present in the same disorder, either concurrently or even separately at different time points. Herein we discuss the neural and cognitive heterogeneity of impulsive and compulsive endophenotypes. These constructs map onto distinct fronto-striatal neural and neurochemical structures interacting both at nodal convergent points and as opponent processes highlighting both the heterogeneity and the commonalities of function. We focus on discoveries made using both translational research methodologies and studies exclusively in humans, and implications for treatment intervention in disorders in which impulsive and compulsive symptoms prevail. We emphasize the relevance of these constructs for understanding dimensional psychiatry.

Optogenetic recruitment of dorsal raphe serotonergic neurons acutely decreases mechanosensory responsivity in behaving mice

The inhibition of sensory responsivity is considered a core serotonin function, yet this hypothesis lacks direct support due to methodological obstacles. We adapted an optogenetic approach to induce acute, robust and specific firing of dorsal raphe serotonergic neurons. In vitro, the responsiveness of individual dorsal raphe serotonergic neurons to trains of light pulses varied with frequency and intensity as well as between cells, and the photostimulation protocol was therefore adjusted to maximize their overall output rate. In vivo, the photoactivation of dorsal raphe serotonergic neurons gave rise to a prominent light-evoked field response that displayed some sensitivity to a 5-HT1A agonist, consistent with autoreceptor inhibition of raphe neurons. In behaving mice, the photostimulation of dorsal raphe serotonergic neurons produced a rapid and reversible decrease in the animals' responses to plantar stimulation, providing a new level of evidence that serotonin gates sensory-driven responses.

Organization of monosynaptic inputs to the serotonin and dopamine neuromodulatory systems

Serotonin and dopamine are major neuromodulators. Here, we used a modified rabies virus to identify monosynaptic inputs to serotonin neurons in the dorsal and median raphe (DR and MR). We found that inputs to DR and MR serotonin neurons are spatially shifted in the forebrain, and MR serotonin neurons receive inputs from more medial structures. Then, we compared these data with inputs to dopamine neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNc). We found that DR serotonin neurons receive inputs from a remarkably similar set of areas as VTA dopamine neurons apart from the striatum, which preferentially targets dopamine neurons. Our results suggest three major input streams: a medial stream regulates MR serotonin neurons, an intermediate stream regulates DR serotonin and VTA dopamine neurons, and a lateral stream regulates SNc dopamine neurons. These results provide fundamental organizational principles of afferent control for serotonin and dopamine.

The effects of lactation on impulsive behavior in vasopressin-deficient Brattleboro rats

Vasopressin (AVP)-deficient Brattleboro rats develop a specific behavioral profile, which-among other things-include altered cognitive performance. This profile is markedly affected by alterations in neuroendocrine state of the animal such as during lactation. Given the links between AVP and cognition we hypothesized that AVP deficiency may lead to changes in impulsivity that is under cognitive control and the changes might be altered by lactation. Comparing virgin and lactating AVP-deficient female Brattleboro rats to their respective controls, we assessed the putative lactation-dependent effects of AVP deficiency on impulsivity in the delay discounting paradigm. Furthermore, to investigate the basis of such effects, we assessed possible interactions of AVP deficiency with GABAergic and serotonergic signaling and stress axis activity, systems playing important roles in impulse control. Our results showed that impulsivity was unaltered by AVP deficiency in virgin rats. In contrast a lactation-induced increase in impulsivity was abolished by AVP deficiency in lactating females. We also found that chlordiazepoxide-induced facilitation of GABAergic and imipramine-induced enhancement of serotonergic activity in virgins led to increased and decreased impulsivity, respectively. In contrast, during lactation these effects were visible only in AVP-deficient rats. These rats also exhibited increased stress axis activity compared to virgin animals, an effect that was abolished by AVP deficiency. Taken together, AVP appears to play a role in the regulation of impulsivity exclusively during lactation: it has an impulsivity increasing effect which is potentially mediated via stress axis-dependent mechanisms and fine-tuning of GABAergic and serotonergic function.

The dorsal raphe nucleus is integral to negative prediction errors in Pavlovian fear

Prediction errors are central to modern learning theories. While brain regions contributing to reward prediction errors have been uncovered, the sources of aversive prediction errors remain largely unknown. Here we used probabilistic and deterministic reinforcement procedures, followed by extinction, to examine the contribution of the dorsal raphe nucleus to negative, aversive prediction errors in Pavlovian fear. Rats with dorsal raphe lesions were able to acquire fear and reduce fear to a non-reinforced deterministic cue. However, dorsal raphe lesions impaired the reduction of fear to a probabilistic cue and fear extinction to a deterministic cue, both of which involve the use of negative prediction errors. The results point to an integral role for the dorsal raphe nucleus in negative prediction error signaling in Pavlovian fear.

Role of Central Serotonin in Anticipation of Rewarding and Punishing Outcomes: Effects of Selective Amygdala or Orbitofrontal 5-HT Depletion

Understanding the role of serotonin (or 5-hydroxytryptamine, 5-HT) in aversive processing has been hampered by the contradictory findings, across studies, of increased sensitivity to punishment in terms of subsequent response choice but decreased sensitivity to punishment-induced response suppression following gross depletion of central 5-HT. To address this apparent discrepancy, the present study determined whether both effects could be found in the same animals by performing localized 5-HT depletions in the amygdala or orbitofrontal cortex (OFC) of a New World monkey, the common marmoset. 5-HT depletion in the amygdala impaired response choice on a probabilistic visual discrimination task by increasing the effectiveness of misleading, or false, punishment and reward, and decreased response suppression in a variable interval test of punishment sensitivity that employed the same reward and punisher. 5-HT depletion in the OFC also disrupted probabilistic discrimination learning and decreased response suppression. Computational modeling of behavior on the discrimination task showed that the lesions reduced reinforcement sensitivity. A novel, unitary account of the findings in terms of the causal role of 5-HT in the anticipation of both negative and positive motivational outcomes is proposed and discussed in relation to current theories of 5-HT function and our understanding of mood and anxiety disorders.

Dorsal raphe neurons signal reward through 5-HT and glutamate

The dorsal raphe nucleus (DRN) in the midbrain is a key center for serotonin (5-hydroxytryptamine; 5-HT)-expressing neurons. Serotonergic neurons in the DRN have been theorized to encode punishment by opposing the reward signaling of dopamine neurons. Here, we show that DRN neurons encode reward, but not punishment, through 5-HT and glutamate. Optogenetic stimulation of DRN Pet-1 neurons reinforces mice to explore the stimulation-coupled spatial region, shifts sucrose preference, drives optical self-stimulation, and directs sensory discrimination learning. DRN Pet-1 neurons increase their firing activity during reward tasks, and this activation can be used to rapidly change neuronal activity patterns in the cortex. Although DRN Pet-1 neurons are often associated with 5-HT, they also release glutamate, and both neurotransmitters contribute to reward signaling. These experiments demonstrate the ability of DRN neurons to organize reward behaviors and might provide insights into the underlying mechanisms of learning facilitation and anhedonia treatment.

Computational modeling of spike generation in serotonergic neurons of the dorsal raphe nucleus

Serotonergic neurons of the dorsal raphe nucleus, with their extensive innervation of limbic and higher brain regions and interactions with the endocrine system have important modulatory or regulatory effects on many cognitive, emotional and physiological processes. They have been strongly implicated in responses to stress and in the occurrence of major depressive disorder and other psychiatric disorders. In order to quantify some of these effects, detailed mathematical models of the activity of such cells are required which describe their complex neurochemistry and neurophysiology. We consider here a single-compartment model of these neurons which is capable of describing many of the known features of spike generation, particularly the slow rhythmic pacemaking activity often observed in these cells in a variety of species. Included in the model are 11 kinds of ion channels: a fast sodium current INa, a delayed rectifier potassium current IKDR, a transient potassium current IA, a slow non-inactivating potassium current IM, a low-threshold calcium current IT, two high threshold calcium currents IL and IN, small and large conductance potassium currents ISK and IBK, a hyperpolarization-activated cation current IH and a leak current ILeak. In Sections 3-8, each current type is considered in detail and parameters estimated from voltage clamp data where possible. Three kinds of model are considered for the BK current and two for the leak current. Intracellular calcium ion concentration Cai is an additional component and calcium dynamics along with buffering and pumping is discussed in Section 9. The remainder of the article contains descriptions of computed solutions which reveal both spontaneous and driven spiking with several parameter sets. Attention is focused on the properties usually associated with these neurons, particularly long duration of action potential, steep upslope on the leading edge of spikes, pacemaker-like spiking, long-lasting afterhyperpolarization and the ramp-like return to threshold after a spike. In some cases the membrane potential trajectories display doublets or have humps or notches as have been reported in some experimental studies. The computed time courses of IA and IT during the interspike interval support the generally held view of a competition between them in influencing the frequency of spiking. Spontaneous activity was facilitated by the presence of IH which has been found in these neurons by some investigators. For reasonable sets of parameters spike frequencies between about 0.6Hz and 1.2Hz are obtained, but frequencies as high as 6Hz could be obtained with special parameter choices. Topics investigated and compared with experiment include shoulders, notches, anodal break phenomena, the effects of noradrenergic input, frequency versus current curves, depolarization block, effects of cell size and the effects of IM. The inhibitory effects of activating 5-HT1A autoreceptors are also investigated. There is a considerable discussion of in vitro versus in vivo firing behavior, with focus on the roles of noradrenergic input, corticotropin-releasing factor and orexinergic inputs. Location of cells within the nucleus is probably a major factor, along with the state of the animal.

Chronic exposure to a predator or its scent does not inhibit male-male competition in male mice lacking brain serotonin

Although it is well-known that defective signaling of the 5-HT system in the brain and stressful stimuli can cause psychological disorders, their combined effects on male–male aggression and sexual attractiveness remain unknown. Our research aimed at examining such effects using tryptophan hydroxylase 2 (Tph2) knockout male mice vs. a rat- or rat scent-based chronic stress model. Tph2^+/+ and Tph2^−/− male mice were placed individually into the rat home cage (rat), a cage containing soiled rat bedding (rat scent) or a cage containing fresh bedding (control) for 5 h every other day for 56 consecutive days. In Tph2^+/+ male mice, rat-exposure decreased male–male aggression and sexual attractiveness of urine odor relative to either rat scent-exposure or control; and rat scent-exposure decreased aggression rather than sexual attractiveness of urine odor compared with control. However, such dose-dependent and long-lasting behavioral inhibitory effects vanished in Tph2^−/− male mice. RT-PCR assay further revealed that putative regulatory genes, such as AR, ERα and GluR4 in the prefrontal cortex, and TrkB-Tc and 5-HTR1A in the hippocampus, were down-regulated at the mRNA level in either rat- or rat scent-exposed Tph2^+/+ male mice, but partially in the Tph2^−/− ones. Hence, we suggest that the dose-dependent and long-lasting inhibitory effects of chronic predator exposure on male–male aggression, sexual attractiveness of urine odor, and mRNA expression of central regulatory genes might be mediated through the 5-HT system in the brain of male mice.

Overlapping neural systems represent cognitive effort and reward anticipation

Anticipating a potential benefit and how difficult it will be to obtain it are valuable skills in a constantly changing environment. In the human brain, the anticipation of reward is encoded by the Anterior Cingulate Cortex (ACC) and Striatum. Naturally, potential rewards have an incentive quality, resulting in a motivational effect improving performance. Recently it has been proposed that an upcoming task requiring effort induces a similar anticipation mechanism as reward, relying on the same cortico-limbic network. However, this overlapping anticipatory activity for reward and effort has only been investigated in a perceptual task. Whether this generalizes to high-level cognitive tasks remains to be investigated. To this end, an fMRI experiment was designed to investigate anticipation of reward and effort in cognitive tasks. A mental arithmetic task was implemented, manipulating effort (difficulty), reward, and delay in reward delivery to control for temporal confounds. The goal was to test for the motivational effect induced by the expectation of bigger reward and higher effort. The results showed that the activation elicited by an upcoming difficult task overlapped with higher reward prospect in the ACC and in the striatum, thus highlighting a pivotal role of this circuit in sustaining motivated behavior.

Neurophysiology of performance monitoring and adaptive behavior

Successful goal-directed behavior requires not only correct action selection, planning, and execution but also the ability to flexibly adapt behavior when performance problems occur or the environment changes. A prerequisite for determining the necessity, type, and magnitude of adjustments is to continuously monitor the course and outcome of one's actions. Feedback-control loops correcting deviations from intended states constitute a basic functional principle of adaptation at all levels of the nervous system. Here, we review the neurophysiology of evaluating action course and outcome with respect to their valence, i.e., reward and punishment, and initiating short- and long-term adaptations, learning, and decisions. Based on studies in humans and other mammals, we outline the physiological principles of performance monitoring and subsequent cognitive, motivational, autonomic, and behavioral adaptation and link them to the underlying neuroanatomy, neurochemistry, psychological theories, and computational models. We provide an overview of invasive and noninvasive systemic measures, such as electrophysiological, neuroimaging, and lesion data. We describe how a wide network of brain areas encompassing frontal cortices, basal ganglia, thalamus, and monoaminergic brain stem nuclei detects and evaluates deviations of actual from predicted states indicating changed action costs or outcomes. This information is used to learn and update stimulus and action values, guide action selection, and recruit adaptive mechanisms that compensate errors and optimize goal achievement.

Optimal indolence: a normative microscopic approach to work and leisure

Dividing limited time between work and leisure when both have their attractions is a common everyday decision. We provide a normative control-theoretic treatment of this decision that bridges economic and psychological accounts. We show how our framework applies to free-operant behavioural experiments in which subjects are required to work (depressing a lever) for sufficient total time (called the price) to receive a reward. When the microscopic benefit-of-leisure increases nonlinearly with duration, the model generates behaviour that qualitatively matches various microfeatures of subjects' choices, including the distribution of leisure bout durations as a function of the pay-off. We relate our model to traditional accounts by deriving macroscopic, molar, quantities from microscopic choices.

Dissociations between interval timing and intertemporal choice following administration of fluoxetine, cocaine, or methamphetamine

The goal of our study was to characterize the relationship between intertemporal choice and interval timing, including determining how drugs that modulate brain serotonin and dopamine levels influence these two processes. In Experiment 1, rats were tested on a standard 40-s peak-interval procedure following administration of fluoxetine (3, 5, or 8 mg/kg) or vehicle to assess basic effects on interval timing. In Experiment 2, rats were tested in a novel behavioral paradigm intended to simultaneously examine interval timing and impulsivity. Rats performed a variant of the bi-peak procedure using 10-s and 40-s target durations with an additional "defection" lever that provided the possibility of a small, immediate reward. Timing functions remained relatively intact, and 'patience' across subjects correlated with peak times, indicating a negative relationship between 'patience' and clock speed. We next examined the effects of fluoxetine (5 mg/kg), cocaine (15 mg/kg), or methamphetamine (1 mg/kg) on task performance. Fluoxetine reduced impulsivity as measured by defection time without corresponding changes in clock speed. In contrast, cocaine and methamphetamine both increased impulsivity and clock speed. Thus, variations in timing may mediate intertemporal choice via dopaminergic inputs. However, a separate, serotonergic system can affect intertemporal choice without affecting interval timing directly. This article is part of a Special Issue entitled: Associative and Temporal Learning.

Serotonin at the nexus of impulsivity and cue reactivity in cocaine addiction

Cocaine abuse and addiction remain great challenges on the public health agendas in the U.S. and the world. Increasingly sophisticated perspectives on addiction to cocaine and other drugs of abuse have evolved with concerted research efforts over the last 30 years. Relapse remains a particularly powerful clinical problem as, even upon termination of drug use and initiation of abstinence, the recidivism rates can be very high. The cycling course of cocaine intake, abstinence and relapse is tied to a multitude of behavioral and cognitive processes including impulsivity (a predisposition toward rapid unplanned reactions to stimuli without regard to the negative consequences), and cocaine cue reactivity (responsivity to cocaine-associated stimuli) cited as two key phenotypes that contribute to relapse vulnerability even years into recovery. Preclinical studies suggest that serotonin (5-hydroxytryptamine; 5-HT) neurotransmission in key neural circuits may contribute to these interlocked phenotypes well as the altered neurobiological states evoked by cocaine that precipitate relapse events. As such, 5-HT is an important target in the quest to understand the neurobiology of relapse-predictive phenotypes, to successfully treat this complex disorder and improve diagnostic and prognostic capabilities. This review emphasizes the role of 5-HT and its receptor proteins in key addiction phenotypes and the implications of current findings to the future of therapeutics in addiction. This article is part of a Special Issue entitled 'NIDA 40th Anniversary Issue'.

The role of the dorsal raphé nucleus in reward-seeking behavior

Pharmacological experiments have shown that the modulation of brain serotonin levels has a strong impact on value-based decision making. Anatomical and physiological evidence also revealed that the dorsal raphé nucleus (DRN), a major source of serotonin, and the dopamine system receive common inputs from brain regions associated with appetitive and aversive information processing. The serotonin and dopamine systems also have reciprocal functional influences on each other. However, the specific mechanism by which serotonin affects value-based decision making is not clear. To understand the information carried by the DRN for reward-seeking behavior, we measured single neuron activity in the primate DRN during the performance of saccade tasks to obtain different amounts of a reward. We found that DRN neuronal activity was characterized by tonic modulation that was altered by the expected and received reward value. Consistent reward-dependent modulation across different task periods suggested that DRN activity kept track of the reward value throughout a trial. The DRN was also characterized by modulation of its activity in the opposite direction by different neuronal subgroups, one firing strongly for the prediction and receipt of large rewards, with the other firing strongly for small rewards. Conversely, putative dopamine neurons showed positive phasic responses to reward-indicating cues and the receipt of an unexpected reward amount, which supports the reward prediction error signal hypothesis of dopamine. I suggest that the tonic reward monitoring signal of the DRN, possibly together with its interaction with the dopamine system, reports a continuous level of motivation throughout the performance of a task. Such a signal may provide "reward context" information to the targets of DRN projections, where it may be integrated further with incoming motivationally salient information.

Trace conditioning in insects-keep the trace!

Trace conditioning is a form of associative learning that can be induced by presenting a conditioned stimulus (CS) and an unconditioned stimulus (US) following each other, but separated by a temporal gap. This gap distinguishes trace conditioning from classical delay conditioning, where the CS and US overlap. To bridge the temporal gap between both stimuli and to form an association between CS and US in trace conditioning, the brain must keep a neural representation of the CS after its termination-a stimulus trace. Behavioral and physiological studies on trace and delay conditioning revealed similarities between the two forms of learning, like similar memory decay and similar odor identity perception in invertebrates. On the other hand differences were reported also, like the requirement of distinct brain structures in vertebrates or disparities in molecular mechanisms in both vertebrates and invertebrates. For example, in commonly used vertebrate conditioning paradigms the hippocampus is necessary for trace but not for delay conditioning, and Drosophila delay conditioning requires the Rutabaga adenylyl cyclase (Rut-AC), which is dispensable in trace conditioning. It is still unknown how the brain encodes CS traces and how they are associated with a US in trace conditioning. Insects serve as powerful models to address the mechanisms underlying trace conditioning, due to their simple brain anatomy, behavioral accessibility and established methods of genetic interference. In this review we summarize the recent progress in insect trace conditioning on the behavioral and physiological level and emphasize similarities and differences compared to delay conditioning. Moreover, we examine proposed molecular and computational models and reassess different experimental approaches used for trace conditioning.

Behavioral and dorsal raphe neuronal activity following acute and chronic methylphenidate in freely behaving rats

Concomitant behavioral and dorsal raphe (DR) neuronal activity were recorded following acute and chronic dose response of methylphenidate (MPD) in freely moving rats previously implanted with permanent semi-microelectrodes using telemetric (wireless) technology. On experimental day (ED) 1, the neuronal and locomotor activity were recorded after saline (baseline) and MPD (0.6, 2.5 or 10.0mg/kg) injection (i.p.). Animals were injected daily with a single dose of MPD for five consecutive days (ED 2-6) to elicit behavioral sensitization or tolerance. After three washout days, the neuronal and locomotor activity recording was resumed on ED 10 followed by saline and MPD rechallenge injection. The main findings were: (1) the same dose of chronic MPD administration elicited behavioral sensitization in some animals and behavioral tolerance in others. (2) 46%, 56% and 73% of DR units responded to acute 0.6, 2.5 and 10.0mg/kg MPD respectively. (3) 89%, 70% and 86% of DR units changed their baseline activity on ED 10 compared to that on ED 1 in the 0.6, 2.5 and 10.0mg/kg MPD groups respectively. (4) A significant difference in ED 10 baseline activity was observed in the DR neuronal population recording from animals expressing behavioral sensitization compared to that of animals expressing behavioral tolerance. (5) 89%, 78% and 88% of DR units responded to chronic 0.6, 2.5 and 10.0mg/kg MPD respectively. (6) The DR neuronal population recording following acute MPD on ED 1 and rechallenge MPD on ED 10 from animals expressing behavioral sensitization was significantly different from the neuronal population recorded from animals exhibited behavioral tolerance. The correlation between the DR neuronal activity and animal's behavior following chronic MPD exposure suggested that the DR neuronal activity may play an important role in the expression of behavioral sensitization and tolerance induced by chronic MPD administration.

A comparison of the subsecond dynamics of neurotransmission of dopamine and serotonin

The neuromodulators dopamine (DA) and serotonin (5-hydroxytryptamine; 5-HT) are similar in a number of ways. Both monoamines can act by volume transmission at metabotropic receptors to modulate synaptic transmission in brain circuits. Presynaptic regulation of 5-HT and DA is governed by parallel processes, and behaviorally, both exert control over emotional processing. However, differences are also apparent: more than twice as many 5-HT receptor subtypes mediate postsynaptic effects than DA receptors and different presynaptic regulation is also emerging. Monoamines are amenable to real-time electrochemical detection using fast scan cyclic voltammetry (FSCV), which allows resolution of the subsecond dynamics of release and reuptake in response to a single action potential. This approach has greatly enriched understanding of DA transmission and has facilitated an integrated view of how DA mediates behavioral control. However, technical challenges are associated with FSCV measurement of 5-HT and understanding of 5-HT transmission at subsecond resolution has not advanced at the same rate. As a result, how the actions of 5-HT at the level of the synapse translate into behavior is poorly understood. Recent technical advances may aid the study of 5-HT in real-time. It is timely, therefore, to compare and contrast what is currently understood of the subsecond characteristics of transmission for DA and 5-HT. In doing so, a number of areas are highlighted as being worthy of exploration for 5-HT.

Dorsal raphe neuronal activities are modulated by methylphenidate

This study investigated the electrophysiological properties of the dorsal raphe nucleus (DR) neurons in response to the acute and repetitive administration of methylphenidate (MPH). Activities of DR neurons were recorded from non-anesthetized, freely behaving rats previously implanted bilaterally with permanent semi microelectrodes. The main findings were: (1) after initial (acute) administration of MPH (2.5 mg/kg i.p.) on experimental day one (ED1), 56 % of DR units significantly changed their firing rates. The majority of the responsive units (88 %) exhibited increased firing rate; (2) daily MPH injections were given on ED2 through ED6 followed by 3 washout days. On ED10, 83 % of the DR units significantly changed their baseline activity compared to the baseline activity on ED1; (3) after rechallenge MPH administration on ED10, 63 % of DR units exhibited significant change in their firing rate; the majority of the responsive units (76 %) exhibited a significant increase in their firing rate; (4) The effect of rechallenge MPH administration on ED10 was compared to the effect of initial MPH on ED1, 47 % DR units exhibited a further significant increase in their firing rate while 53 % DR units exhibited decrease or non-change in their firing rate which can be interpreted as electrophysiological sensitization or tolerance. In conclusion, this study demonstrated that acute MPH administration modulated the DR neuronal activities. Repetitive MPH administration modulated the baseline activities of DR units and elicited neurophysiological sensitization or tolerance. The results indicated that MPH affects DR neuronal activity.

Neurons in monkey dorsal raphe nucleus code beginning and progress of step-by-step schedule, reward expectation, and amount of reward outcome in the reward schedule task

The dorsal raphe nucleus is the major source of serotonin in the brain. It is connected to brain regions related to reward processing, and the neurons show activity related to predicted reward outcome. Clinical observations also suggest that it is important in maintaining alertness and its apparent role in addiction seems to be related to reward processing. Here, we examined whether the neurons in dorsal raphe carry signals about reward outcome and task progress during multitrial schedules. We recorded from 98 single neurons in dorsal raphe of two monkeys. The monkeys perform one, two, or three visual discrimination trials (schedule), obtaining one, two, or three drops of liquid. In the valid cue condition, the length and brightness of a visual cue indicated schedule progress and reward amount, respectively. In the random cue condition, the visual cue was randomly presented with respect to schedule length and reward amount. We found information encoded about (1) schedule onset, (2) reward expectation, (3) reward outcome, and (4) reward amount in the mean firing rates. Information theoretic analysis showed that the temporal variation of the neuronal responses contained additional information related to the progress of the schedule toward the reward rather than only discriminating schedule onset or reward/no reward. When considered in light of all that is known about the raphe in anatomy, physiology, and behavior, the rich encoding about both task progress and predicted reward outcome makes the raphe a strong candidate for providing signals throughout the brain to coordinate persistent goal-seeking behavior.

Reinforcement learning using a continuous time actor-critic framework with spiking neurons

Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.

As every dog owner knows, animals repeat behaviors that earn them rewards. But what is the brain machinery that underlies this reward-based learning? Experimental research points to plasticity of the synaptic connections between neurons, with an important role played by the neuromodulator dopamine, but the exact way synaptic activity and neuromodulation interact during learning is not precisely understood. Here we propose a model explaining how reward signals might interplay with synaptic plasticity, and use the model to solve a simulated maze navigation task. Our model extends an idea from the theory of reinforcement learning: one group of neurons form an “actor,” responsible for choosing the direction of motion of the animal. Another group of neurons, the “critic,” whose role is to predict the rewards the actor will gain, uses the mismatch between actual and expected reward to teach the synapses feeding both groups. Our learning agent learns to reliably navigate its maze to find the reward. Remarkably, the synaptic learning rule that we derive from theoretical considerations is similar to previous rules based on experimental evidence.