Reinforcement signaling of punishment versus relief in fruit flies

Prediction error drives associative learning and conditioned behavior in a spiking model of Drosophila larva

Predicting reinforcement from sensory cues is beneficial for goal-directed behavior. In insect brains, underlying associations between cues and reinforcement, encoded by dopaminergic neurons, are formed in the mushroom body. We propose a spiking model of the Drosophila larva mushroom body. It includes a feedback motif conveying learned reinforcement expectation to dopaminergic neurons, which can compute prediction error as the difference between expected and present reinforcement. We demonstrate that this can serve as a driving force in learning. When combined with synaptic homeostasis, our model accounts for theoretically derived features of acquisition and loss of associations that depend on the intensity of the reinforcement and its temporal proximity to the cue. From modeling olfactory learning over the time course of behavioral experiments and simulating the locomotion of individual larvae toward or away from odor sources in a virtual environment, we conclude that learning driven by prediction errors can explain larval behavior.

Rewarding Capacity of Optogenetically Activating a Giant GABAergic Central-Brain Interneuron in Larval Drosophila

Larvae of the fruit fly Drosophila melanogaster are a powerful study case for understanding the neural circuits underlying behavior. Indeed, the numerical simplicity of the larval brain has permitted the reconstruction of its synaptic connectome, and genetic tools for manipulating single, identified neurons allow neural circuit function to be investigated with relative ease and precision. We focus on one of the most complex neurons in the brain of the larva (of either sex), the GABAergic anterior paired lateral neuron (APL). Using behavioral and connectomic analyses, optogenetics, Ca2+ imaging, and pharmacology, we study how APL affects associative olfactory memory. We first provide a detailed account of the structure, regional polarity, connectivity, and metamorphic development of APL, and further confirm that optogenetic activation of APL has an inhibiting effect on its main targets, the mushroom body Kenyon cells. All these findings are consistent with the previously identified function of APL in the sparsening of sensory representations. To our surprise, however, we found that optogenetically activating APL can also have a strong rewarding effect. Specifically, APL activation together with odor presentation establishes an odor-specific, appetitive, associative short-term memory, whereas naive olfactory behavior remains unaffected. An acute, systemic inhibition of dopamine synthesis as well as an ablation of the dopaminergic pPAM neurons impair reward learning through APL activation. Our findings provide a study case of complex circuit function in a numerically simple brain, and suggest a previously unrecognized capacity of central-brain GABAergic neurons to engage in dopaminergic reinforcement.SIGNIFICANCE STATEMENT The single, identified giant anterior paired lateral (APL) neuron is one of the most complex neurons in the insect brain. It is GABAergic and contributes to the sparsening of neuronal activity in the mushroom body, the memory center of insects. We provide the most detailed account yet of the structure of APL in larval Drosophila as a neurogenetically accessible study case. We further reveal that, contrary to expectations, the experimental activation of APL can exert a rewarding effect, likely via dopaminergic reward pathways. The present study both provides an example of unexpected circuit complexity in a numerically simple brain, and reports an unexpected effect of activity in central-brain GABAergic circuits.

Metamorphosis of memory circuits in Drosophila reveals a strategy for evolving a larval brain

Mushroom bodies (MB) of adult Drosophila have a core of thousands of Kenyon neurons; axons of the early-born g class form a medial lobe and those from later-born α'β' and αβ classes form both medial and vertical lobes. The larva, however, hatches with only γ neurons and forms a vertical lobe 'facsimile' using larval-specific axon branches from its γ neurons. MB input (MBINs) and output (MBONs) neurons divide the Kenyon neuron lobes into discrete computational compartments. The larva has 10 such compartments while the adult has 16. We determined the fates of 28 of the 32 MBONs and MBINs that define the 10 larval compartments. Seven compartments are subsequently incorporated into the adult MB; four of their MBINs die, while 12 MBINs/MBONs remodel to function in adult compartments. The remaining three compartments are larval specific. At metamorphosis their MBIN/MBONs trans-differentiate, leaving the MB for other adult brain circuits. The adult vertical lobes are made de novo using MBONs/MBINs recruited from pools of adult-specific neurons. The combination of cell death, compartment shifting, trans-differentiation, and recruitment of new neurons result in no larval MBIN-MBON connections being maintained through metamorphosis. At this simple level, then, we find no anatomical substrate for a memory trace persisting from larva to adult. The adult phenotype of the trans-differentiating neurons represents their evolutionarily ancestral phenotype while their larval phenotype is a derived adaptation for the larval stage. These cells arise primarily within lineages that also produce permanent MBINs and MBONs, suggesting that larval specifying factors may allow information related to birth-order or sibling identity to be interpreted in a modified manner in the larva to allow these neurons to acquire larval phenotypic modifications. The loss of such factors at metamorphosis then allows these neurons to revert to their ancestral functions in the adult.

Optogenetically induced reward and 'frustration' memory in larval Drosophila

Humans and animals alike form oppositely valenced memories for stimuli that predict the occurrence versus the termination of a reward: appetitive 'reward' memory for stimuli associated with the occurrence of a reward and aversive 'frustration' memory for stimuli that are associated with its termination. We characterize these memories in larval Drosophila using a combination of Pavlovian conditioning, optogenetic activation of the dopaminergic central-brain DAN-i1864 neuron, and high-resolution video-tracking. This reveals their dependency on the number of training trials and the duration of DAN-i1864 activation, their temporal stability, and the parameters of locomotion that are modulated during memory expression. Together with previous results on 'punishment' versus 'relief' learning by DAN-f1 neuron activation, this reveals a 2x2 matrix of timing-dependent memory valence for the occurrence/ termination of reward/ punishment. These findings should aid the understanding and modelling of how brains decipher the predictive, causal structure of events around a target reinforcing occurrence.

Dopamine activity in projection neurons regulates short-lasting olfactory approach memory in Drosophila

Survival in many animals requires the ability to associate certain cues with danger and others with safety. In a Drosophila melanogaster aversive olfactory conditioning paradigm, flies are exposed to two odours, one presented coincidentally with electrical shocks, and a second presented 45 s after shock cessation. When flies are later given a choice between these two odours, they avoid the shock‐paired odour and prefer the unpaired odour. While many studies have examined how flies learn to avoid the shock‐paired odour through formation of odour‐fear associations, here we demonstrate that conditioning also causes flies to actively approach the second odour. In contrast to fear memories, which are longer lasting and requires activity of D1‐like dopamine receptors only in the mushroom bodies, approach memory is short‐lasting and requires activity of D1‐like dopamine receptors in projection neurons originating from the antennal lobes, primary olfactory centers. Further, while recall of fear memories requires activity of the mushroom bodies, recall of approach memories does not. Our data suggest that olfactory approach memory is formed using different mechanisms in different brain locations compared to aversive and appetitive olfactory memories.

Honey bees can store and retrieve independent memory traces after complex experiences that combine appetitive and aversive associations

Real-world experiences do often mix appetitive and aversive events. Understanding the ability of animals to extract, store and use this information is an important issue in neurobiology. We used honey bees as model organism to study learning and memory after a differential conditioning that combines appetitive and aversive training trials. First of all, we describe an aversive conditioning paradigm that constitutes a clear opposite of the well known appetitive olfactory conditioning of the proboscis extension response. A neutral odour is presented paired with the bitter substance quinine. Aversive memory is evidenced later as an odour-specific impairment in appetitive conditioning. Then we tested the effect of mixing appetitive and aversive conditioning trials distributed along the same training session. Differential conditioning protocols like this were used before to study the ability to discriminate odours, however they were not focused on whether appetitive and aversive memories are formed. We found that after a differential conditioning, honey bees establish independent appetitive and aversive memories that do not interfere with each other during acquisition or storage. Finally, we moved the question forward to retrieval and memory expression to evaluate what happens when appetitive and the aversive learned odours are mixed during test. Interestingly, opposite memories compete in a way that they do not cancel each other out. Honey bees showed the ability to switch from expressing appetitive to aversive memory depending on their satiation level.

Associative learning drives longitudinally graded presynaptic plasticity of neurotransmitter release along axonal compartments

Anatomical and physiological compartmentalization of neurons is a mechanism to increase the computational capacity of a circuit, and a major question is what role axonal compartmentalization plays. Axonal compartmentalization may enable localized, presynaptic plasticity to alter neuronal output in a flexible, experience-dependent manner. Here, we show that olfactory learning generates compartmentalized, bidirectional plasticity of acetylcholine release that varies across the longitudinal compartments of Drosophila mushroom body (MB) axons. The directionality of the learning-induced plasticity depends on the valence of the learning event (aversive vs. appetitive), varies linearly across proximal to distal compartments following appetitive conditioning, and correlates with learning-induced changes in downstream mushroom body output neurons (MBONs) that modulate behavioral action selection. Potentiation of acetylcholine release was dependent on the CaV2.1 calcium channel subunit cacophony. In addition, contrast between the positive conditioned stimulus and other odors required the inositol triphosphate receptor, which maintained responsivity to odors upon repeated presentations, preventing adaptation. Downstream from the MB, a set of MBONs that receive their input from the γ3 MB compartment were required for normal appetitive learning, suggesting that they represent a key node through which reward learning influences decision-making. These data demonstrate that learning drives valence-correlated, compartmentalized, bidirectional potentiation, and depression of synaptic neurotransmitter release, which rely on distinct mechanisms and are distributed across axonal compartments in a learning circuit.

Associative learning in larval and adult Drosophila is impaired by the dopamine-synthesis inhibitor 3-Iodo-L-tyrosine

Across the animal kingdom, dopamine plays a crucial role in conferring reinforcement signals that teach animals about the causal structure of the world. In the fruit fly Drosophila melanogaster, dopaminergic reinforcement has largely been studied using genetics, whereas pharmacological approaches have received less attention. Here, we apply the dopamine-synthesis inhibitor 3-Iodo-L-tyrosine (3IY), which causes acute systemic inhibition of dopamine signaling, and investigate its effects on Pavlovian conditioning. We find that 3IY feeding impairs sugar-reward learning in larvae while leaving task-relevant behavioral faculties intact, and that additional feeding of a precursor of dopamine (L-3,4-dihydroxyphenylalanine, L-DOPA), rescues this impairment. Concerning a different developmental stage and for the aversive valence domain. Moreover, we demonstrate that punishment learning by activating the dopaminergic neuron PPL1-γ1pedc in adult flies is also impaired by 3IY feeding, and can likewise be rescued by L-DOPA. Our findings exemplify the advantages of using a pharmacological approach in combination with the genetic techniques available in D. melanogaster to manipulate neuronal and behavioral function.

Summary: We surveyed the effects of a dopamine-synthesis inhibitor on associative learning in larval and adult Drosophila. This approach can supplement genetic tools in investigating the conserved reinforcing function of dopamine.

The Role of Dopamine in Associative Learning in Drosophila: An Updated Unified Model

Learning to associate a positive or negative experience with an unrelated cue after the presentation of a reward or a punishment defines associative learning. The ability to form associative memories has been reported in animal species as complex as humans and as simple as insects and sea slugs. Associative memory has even been reported in tardigrades [1], species that diverged from other animal phyla 500 million years ago. Understanding the mechanisms of memory formation is a fundamental goal of neuroscience research. In this article, we work on resolving the current contradictions between different Drosophila associative memory circuit models and propose an updated version of the circuit model that predicts known memory behaviors that current models do not. Finally, we propose a model for how dopamine may function as a reward prediction error signal in Drosophila, a dopamine function that is well-established in mammals but not in insects [2, 3].

Perception of repeated pain relief with controllable and uncontrollable pain

BACKGROUND

The ultimate goal of pain research is to provide effective routes for pain relief. Nevertheless, the perception pain relief as a change in pain intensity and un-/pleasantness has only been rarely investigated. It has been demonstrated that pain relief has rewarding and reinforcing properties, but it remains unknown whether the perception of pain relief changes when pain reductions occur repeatedly. Further, it remains an open question whether the perception of pain relief depends on the controllability of the preceding pain.

METHODS

In this study, healthy volunteers (N = 38) received five cycles of painful heat stimulation and reduction of this stimulation to a non-painful warm stimulation once in a condition with control of the stimulation and once without control. Participants rated perceived intensity and un-/pleasantness on visual analogue scales during the heat stimulation and immediately after its reduction.

RESULTS

Results showed that perceived pain relief, estimated by the difference in ratings during ongoing heat stimulation and after its reduction, increased with repetitions. However, this increase levelled off after two to four repetitions. Further, perceived pain relief was larger in the condition without control compared to the condition with control.

CONCLUSION

The perception of pain relief can be modulated similar to the perception of pain by stimulus characteristics and psychological factors. Mechanistic knowledge about such modulating factors is important, because they can determine, e.g., the amount of requested pain killers in clinical settings and the efficacy of pain relief as a reinforcing stimulus.

SIGNIFICANCE

When in pain, pain relief can become an all-dominate goal. The perception of such pain relief can vary depending on external and internal characteristics and thus modulate, e.g., requests for pain killers in clinical settings. Here, we show that perceived intensity and pleasantness of pain relief changes with repetitions and whether the preceding pain is perceived as uncontrollable. Such mechanistic knowledge needs to be considered to maximize the effects of pain relief as a rewarding and reinforcing stimulus.

Aversive teaching signals from individual dopamine neurons in larval Drosophila show qualitative differences in their temporal "fingerprint"

Dopamine serves many functions, and dopamine neurons are correspondingly diverse. We use a combination of optogenetics, behavioral experiments, and high-resolution video-tracking to probe for the functional capacities of two single, identified dopamine neurons in larval Drosophila. The DAN-f1 and the DAN-d1 neuron were recently found to carry aversive teaching signals during Pavlovian olfactory learning. We enquire into a fundamental feature of these teaching signals, namely their temporal "fingerprint". That is, receiving punishment feels bad, whereas being relieved from it feels good, and animals and humans alike learn with opposite valence about the occurrence and the termination of punishment (the same principle applies in the appetitive domain, with opposite sign). We find that DAN-f1 but not DAN-d1 can mediate such timing-dependent valence reversal: presenting an odor before DAN-f1 activation leads to learned avoidance of the odor (punishment memory), whereas presenting the odor upon termination of DAN-f1 activation leads to learned approach (relief memory). In contrast, DAN-d1 confers punishment memory only. These effects are further characterized in terms of the impact of the duration of optogenetic activation, the temporal stability of the memories thus established, and the specific microbehavioral patterns of locomotion through which they are expressed. Together with recent findings in the appetitive domain and from adult Drosophila, our results suggest that heterogeneity in the temporal fingerprint of teaching signals might be a more general principle of reinforcement processing through dopamine neurons.

Predictive olfactory learning in Drosophila

Olfactory learning and conditioning in the fruit fly is typically modelled by correlation-based associative synaptic plasticity. It was shown that the conditioning of an odor-evoked response by a shock depends on the connections from Kenyon cells (KC) to mushroom body output neurons (MBONs). Although on the behavioral level conditioning is recognized to be predictive, it remains unclear how MBONs form predictions of aversive or appetitive values (valences) of odors on the circuit level. We present behavioral experiments that are not well explained by associative plasticity between conditioned and unconditioned stimuli, and we suggest two alternative models for how predictions can be formed. In error-driven predictive plasticity, dopaminergic neurons (DANs) represent the error between the predictive odor value and the shock strength. In target-driven predictive plasticity, the DANs represent the target for the predictive MBON activity. Predictive plasticity in KC-to-MBON synapses can also explain trace-conditioning, the valence-dependent sign switch in plasticity, and the observed novelty-familiarity representation. The model offers a framework to dissect MBON circuits and interpret DAN activity during olfactory learning.

Spaced Training Forms Complementary Long-Term Memories of Opposite Valence in Drosophila

Forming long-term memory (LTM) often requires repetitive experience spread over time. Studies in Drosophila suggest aversive olfactory LTM is optimal after spaced training, multiple trials of differential odor conditioning with rest intervals. Memory after spaced training is frequently compared to that after the same number of trials without intervals. Here we show that, after spaced training, flies acquire additional information and form an aversive memory for the shock-paired odor and a slowly emerging and more persistent "safety-memory" for the explicitly unpaired odor. Safety-memory acquisition requires repetition, order, and spacing of the training trials and relies on triggering specific rewarding dopaminergic neurons. Co-existence of aversive and safety memories is evident as depression of odor-specific responses at different combinations of junctions in the mushroom body output network; combining two outputs appears to signal relative safety. Having complementary aversive and safety memories augments LTM performance after spaced training by making the odor preference more certain.

Visualization of a Distributed Synaptic Memory Code in the Drosophila Brain

During associative conditioning, animals learn which sensory cues are predictive for positive or negative conditions. Because sensory cues are encoded by distributed neurons, one has to monitor plasticity across many synapses to capture how learned information is encoded. We analyzed synaptic boutons of Kenyon cells of the Drosophila mushroom body γ lobe, a brain structure that mediates olfactory learning. A fluorescent Ca²⁺ sensor was expressed in single Kenyon cells so that axonal boutons could be assigned to distinct cells and Ca²⁺ could be measured across many animals. Learning induced directed synaptic plasticity in specific compartments along the axons. Moreover, we show that odor-evoked Ca²⁺ dynamics across boutons decorrelate as a result of associative learning. Information theory indicates that learning renders the stimulus representation more distinct compared with naive stimuli. These data reveal that synaptic boutons rather than cells act as individually modifiable units, and coherence among them is a memory-encoding parameter.

Toward nanoscale localization of memory engrams in Drosophila

The Mushroom Body (MB) is the primary location of stored associative memories in the Drosophila brain. We discuss recent advances in understanding the MB's neuronal circuits made using advanced light microscopic methods and cell-type-specific genetic tools. We also review how the compartmentalized nature of the MB's organization allows this brain area to form and store memories with widely different dynamics.

The Unc13A isoform is important for phasic release and olfactory memory formation at mushroom body synapses

The cellular analysis of mushroom body (MB)-dependent memory forming processes is far advanced, whereas, the molecular and physiological understanding of their synaptic basis lags behind. Recent analysis of the Drosophila olfactory system showed that Unc13A, a member of the M(Unc13) release factor family, promotes a phasic, high release probability component, while Unc13B supports a slower tonic release component, reflecting their different nanoscopic positioning within individual active zones. We here use STED super-resolution microscopy of MB lobe synapses to show that Unc13A clusters closer to the active zone centre than Unc13B. Unc13A specifically supported phasic transmission and short-term plasticity of Kenyon cell:output neuron synapses, measured by combining electrophysiological recordings of output neurons with optogenetic stimulation. Knockdown of unc13A within Kenyon cells provoked drastic deficits of olfactory aversive short-term and anaesthesia-sensitive middle-term memory. Knockdown of unc13B provoked milder memory deficits. Thus, a low frequency domain transmission component is probably crucial for the proper representation of memory-associated activity patterns, consistent with sparse Kenyon cell activation during memory acquisition and retrieval. Notably, Unc13A/B ratios appeared highly diversified across MB lobes, leaving room for an interplay of activity components in memory encoding and retrieval.

An optogenetic analogue of second-order reinforcement in Drosophila

In insects, odours are coded by the combinatorial activation of ascending pathways, including their third-order representation in mushroom body Kenyon cells. Kenyon cells also receive intersecting input from ascending and mostly dopaminergic reinforcement pathways. Indeed, in Drosophila, presenting an odour together with activation of the dopaminergic mushroom body input neuron PPL1-01 leads to a weakening of the synapse between Kenyon cells and the approach-promoting mushroom body output neuron MBON-11. As a result of such weakened approach tendencies, flies avoid the shock-predicting odour in a subsequent choice test. Thus, increased activity in PPL1-01 stands for punishment, whereas reduced activity in MBON-11 stands for predicted punishment. Given that punishment-predictors can themselves serve as punishments of second order, we tested whether presenting an odour together with the optogenetic silencing of MBON-11 would lead to learned odour avoidance, and found this to be the case. In turn, the optogenetic activation of MBON-11 together with odour presentation led to learned odour approach. Thus, manipulating activity in MBON-11 can be an analogue of predicted, second-order reinforcement.

Distinct Dopamine Receptor Pathways Underlie the Temporal Sensitivity of Associative Learning

Animals rely on the relative timing of events in their environment to form and update predictive associations, but the molecular and circuit mechanisms for this temporal sensitivity remain incompletely understood. Here, we show that olfactory associations in Drosophila can be written and reversed on a trial-by-trial basis depending on the temporal relationship between an odor cue and dopaminergic reinforcement. Through the synchronous recording of neural activity and behavior, we show that reversals in learned odor attraction correlate with bidirectional neural plasticity in the mushroom body, the associative olfactory center of the fly. Two dopamine receptors, DopR1 and DopR2, contribute to this temporal sensitivity by coupling to distinct second messengers and directing either synaptic depression or potentiation. Our results reveal how dopamine-receptor signaling pathways can detect the order of events to instruct opposing forms of synaptic and behavioral plasticity, allowing animals to flexibly update their associations in a dynamic environment.

Mapping Odor to Action: (Dopaminergic) Timing Is Everything

Animal brains use the relative timing between sensory cues and behaviorally salient events to form predictive associations about their environment. Handler and colleagues provide new mechanistic insights into how differential signaling downstream of dopamine receptors couples this timing to the dynamic reweighting of synapses that link sensation to action.

One-trial learning in larval Drosophila

Animals of many species are capable of “small data” learning, that is, of learning without repetition. Here we introduce larval Drosophila melanogaster as a relatively simple study case for such one-trial learning. Using odor-food associative conditioning, we first show that a sugar that is both sweet and nutritious (fructose) and sugars that are only sweet (arabinose) or only nutritious (sorbitol) all support appetitive one-trial learning. The same is the case for the optogenetic activation of a subset of dopaminergic neurons innervating the mushroom body, the memory center of the insects. In contrast, no one-trial learning is observed for an amino acid reward (aspartic acid). As regards the aversive domain, one-trial learning is demonstrated for high-concentration sodium chloride, but is not observed for a bitter tastant (quinine). Second, we provide follow-up, parametric analyses of odor-fructose learning. Specifically, we ascertain its dependency on the number and duration of training trials, the requirements for the behavioral expression of one-trial odor-fructose memory, its temporal stability, and the feasibility of one-trial differential conditioning. Our results set the stage for a neurogenetic analysis of one-trial learning and define the requirements for modeling mnemonic processes in the larva.