Models of heterogeneous dopamine signaling in an insect learning and memory center

A feature-specific prediction error model explains dopaminergic heterogeneity

The hypothesis that midbrain dopamine (DA) neurons broadcast a reward prediction error (RPE) is among the great successes of computational neuroscience. However, recent results contradict a core aspect of this theory: specifically that the neurons convey a scalar, homogeneous signal. While the predominant family of extensions to the RPE model replicates the classic model in multiple parallel circuits, we argue that these models are ill suited to explain reports of heterogeneity in task variable encoding across DA neurons. Instead, we introduce a complementary 'feature-specific RPE' model, positing that individual ventral tegmental area DA neurons report RPEs for different aspects of an animal's moment-to-moment situation. Further, we show how our framework can be extended to explain patterns of heterogeneity in action responses reported among substantia nigra pars compacta DA neurons. This theory reconciles new observations of DA heterogeneity with classic ideas about RPE coding while also providing a new perspective of how the brain performs reinforcement learning in high-dimensional environments.

Beyond prediction error: 25 years of modeling the associations formed in the insect mushroom body

The insect mushroom body has gained increasing attention as a system in which the computational basis of neural learning circuits can be unraveled. We now understand in detail the key locations in this circuit where synaptic associations are formed between sensory patterns and values leading to actions. However, the actual learning rule (or rules) implemented by neural activity and leading to synaptic change is still an open question. Here, I survey the diversity of answers that have been offered in computational models of this system over the past decades, including the recurring assumption-in line with top-down theories of associative learning-that the core function is to reduce prediction error. However, I will argue, a more bottom-up approach may ultimately reveal a richer algorithmic capacity in this still enigmatic brain neuropil.

Roles of feedback and feed-forward networks of dopamine subsystems: insights from Drosophila studies

Across animal species, dopamine-operated memory systems comprise anatomically segregated, functionally diverse subsystems. Although individual subsystems could operate independently to support distinct types of memory, the logical interplay between subsystems is expected to enable more complex memory processing by allowing existing memory to influence future learning. Recent comprehensive ultrastructural analysis of the Drosophila mushroom body revealed intricate networks interconnecting the dopamine subsystems-the mushroom body compartments. Here, we review the functions of some of these connections that are beginning to be understood. Memory consolidation is mediated by two different forms of network: A recurrent feedback loop within a compartment maintains sustained dopamine activity required for consolidation, whereas feed-forward connections across compartments allow short-term memory formation in one compartment to open the gate for long-term memory formation in another compartment. Extinction and reversal of aversive memory rely on a similar feed-forward circuit motif that signals omission of punishment as a reward, which triggers plasticity that counteracts the original aversive memory trace. Finally, indirect feed-forward connections from a long-term memory compartment to short-term memory compartments mediate higher-order conditioning. Collectively, these emerging studies indicate that feedback control and hierarchical connectivity allow the dopamine subsystems to work cooperatively to support diverse and complex forms of learning.

Specific connectivity optimizes learning in thalamocortical loops

Thalamocortical loops have a central role in cognition and motor control, but precisely how they contribute to these processes is unclear. Recent studies showing evidence of plasticity in thalamocortical synapses indicate a role for the thalamus in shaping cortical dynamics through learning. Since signals undergo a compression from the cortex to the thalamus, we hypothesized that the computational role of the thalamus depends critically on the structure of corticothalamic connectivity. To test this, we identified the optimal corticothalamic structure that promotes biologically plausible learning in thalamocortical synapses. We found that corticothalamic projections specialized to communicate an efference copy of the cortical output benefit motor control, while communicating the modes of highest variance is optimal for working memory tasks. We analyzed neural recordings from mice performing grasping and delayed discrimination tasks and found corticothalamic communication consistent with these predictions. These results suggest that the thalamus orchestrates cortical dynamics in a functionally precise manner through structured connectivity.

Data-driven multiscale computational models of cortical and subcortical regions

Data-driven computational models of neurons, synapses, microcircuits, and mesocircuits have become essential tools in modern brain research. The goal of these multiscale models is to integrate and synthesize information from different levels of brain organization, from cellular properties, dendritic excitability, and synaptic dynamics to microcircuits, mesocircuits, and ultimately behavior. This article surveys recent advances in the genesis of data-driven computational models of mammalian neural networks in cortical and subcortical areas. I discuss the challenges and opportunities in developing data-driven multiscale models, including the need for interdisciplinary collaborations, the importance of model validation and comparison, and the potential impact on basic and translational neuroscience research. Finally, I highlight future directions and emerging technologies that will enable more comprehensive and predictive data-driven models of brain function and dysfunction.

Prediction error drives associative learning and conditioned behavior in a spiking model of Drosophila larva

Predicting reinforcement from sensory cues is beneficial for goal-directed behavior. In insect brains, underlying associations between cues and reinforcement, encoded by dopaminergic neurons, are formed in the mushroom body. We propose a spiking model of the Drosophila larva mushroom body. It includes a feedback motif conveying learned reinforcement expectation to dopaminergic neurons, which can compute prediction error as the difference between expected and present reinforcement. We demonstrate that this can serve as a driving force in learning. When combined with synaptic homeostasis, our model accounts for theoretically derived features of acquisition and loss of associations that depend on the intensity of the reinforcement and its temporal proximity to the cue. From modeling olfactory learning over the time course of behavioral experiments and simulating the locomotion of individual larvae toward or away from odor sources in a virtual environment, we conclude that learning driven by prediction errors can explain larval behavior.

Minimal circuit motifs for second-order conditioning in the insect mushroom body

In well-established first-order conditioning experiments, the concurrence of a sensory cue with reinforcement forms an association, allowing the cue to predict future reinforcement. In the insect mushroom body, a brain region central to learning and memory, such associations are encoded in the synapses between its intrinsic and output neurons. This process is mediated by the activity of dopaminergic neurons that encode reinforcement signals. In second-order conditioning, a new sensory cue is paired with an already established one that presumably activates dopaminergic neurons due to its predictive power of the reinforcement. We explored minimal circuit motifs in the mushroom body for their ability to support second-order conditioning using mechanistic models. We found that dopaminergic neurons can either be activated directly by the mushroom body's intrinsic neurons or via feedback from the output neurons via several pathways. We demonstrated that the circuit motifs differ in their computational efficiency and robustness. Beyond previous research, we suggest an additional motif that relies on feedforward input of the mushroom body intrinsic neurons to dopaminergic neurons as a promising candidate for experimental evaluation. It differentiates well between trained and novel stimuli, demonstrating robust performance across a range of model parameters.

Reward expectations direct learning and drive operant matching in Drosophila

Foraging animals must use decision-making strategies that dynamically adapt to the changing availability of rewards in the environment. A wide diversity of animals do this by distributing their choices in proportion to the rewards received from each option, Herrnstein's operant matching law. Theoretical work suggests an elegant mechanistic explanation for this ubiquitous behavior, as operant matching follows automatically from simple synaptic plasticity rules acting within behaviorally relevant neural circuits. However, no past work has mapped operant matching onto plasticity mechanisms in the brain, leaving the biological relevance of the theory unclear. Here, we discovered operant matching in Drosophila and showed that it requires synaptic plasticity that acts in the mushroom body and incorporates the expectation of reward. We began by developing a dynamic foraging paradigm to measure choices from individual flies as they learn to associate odor cues with probabilistic rewards. We then built a model of the fly mushroom body to explain each fly's sequential choice behavior using a family of biologically realistic synaptic plasticity rules. As predicted by past theoretical work, we found that synaptic plasticity rules could explain fly matching behavior by incorporating stimulus expectations, reward expectations, or both. However, by optogenetically bypassing the representation of reward expectation, we abolished matching behavior and showed that the plasticity rule must specifically incorporate reward expectations. Altogether, these results reveal the first synapse-level mechanisms of operant matching and provide compelling evidence for the role of reward expectation signals in the fly brain.

Modulation of aversive value coding in the vertebrate and invertebrate brain

Avoiding potentially dangerous situations is key for the survival of any organism. Throughout life, animals learn to avoid environments, stimuli or actions that can lead to bodily harm. While the neural bases for appetitive learning, evaluation and value-based decision-making have received much attention, recent studies have revealed more complex computations for aversive signals during learning and decision-making than previously thought. Furthermore, previous experience, internal state and systems level appetitive-aversive interactions seem crucial for learning specific aversive value signals and making appropriate choices. The emergence of novel methodologies (computation analysis coupled with large-scale neuronal recordings, neuronal manipulations at unprecedented resolution offered by genetics, viral strategies and connectomics) has helped to provide novel circuit-based models for aversive (and appetitive) valuation. In this review, we focus on recent vertebrate and invertebrate studies yielding strong evidence that aversive value information can be computed by a multitude of interacting brain regions, and that past experience can modulate future aversive learning and therefore influence value-based decisions.

Hierarchical architecture of dopaminergic circuits enables second-order conditioning in Drosophila

Dopaminergic neurons with distinct projection patterns and physiological properties compose memory subsystems in a brain. However, it is poorly understood whether or how they interact during complex learning. Here, we identify a feedforward circuit formed between dopamine subsystems and show that it is essential for second-order conditioning, an ethologically important form of higher-order associative learning. The Drosophila mushroom body comprises a series of dopaminergic compartments, each of which exhibits distinct memory dynamics. We find that a slow and stable memory compartment can serve as an effective 'teacher' by instructing other faster and transient memory compartments via a single key interneuron, which we identify by connectome analysis and neurotransmitter prediction. This excitatory interneuron acquires enhanced response to reward-predicting odor after first-order conditioning and, upon activation, evokes dopamine release in the 'student' compartments. These hierarchical connections between dopamine subsystems explain distinct properties of first- and second-order memory long known by behavioral psychologists.

Associative learning drives longitudinally graded presynaptic plasticity of neurotransmitter release along axonal compartments

Anatomical and physiological compartmentalization of neurons is a mechanism to increase the computational capacity of a circuit, and a major question is what role axonal compartmentalization plays. Axonal compartmentalization may enable localized, presynaptic plasticity to alter neuronal output in a flexible, experience-dependent manner. Here, we show that olfactory learning generates compartmentalized, bidirectional plasticity of acetylcholine release that varies across the longitudinal compartments of Drosophila mushroom body (MB) axons. The directionality of the learning-induced plasticity depends on the valence of the learning event (aversive vs. appetitive), varies linearly across proximal to distal compartments following appetitive conditioning, and correlates with learning-induced changes in downstream mushroom body output neurons (MBONs) that modulate behavioral action selection. Potentiation of acetylcholine release was dependent on the Ca_V2.1 calcium channel subunit cacophony. In addition, contrast between the positive conditioned stimulus and other odors required the inositol triphosphate receptor, which maintained responsivity to odors upon repeated presentations, preventing adaptation. Downstream from the MB, a set of MBONs that receive their input from the γ3 MB compartment were required for normal appetitive learning, suggesting that they represent a key node through which reward learning influences decision-making. These data demonstrate that learning drives valence-correlated, compartmentalized, bidirectional potentiation, and depression of synaptic neurotransmitter release, which rely on distinct mechanisms and are distributed across axonal compartments in a learning circuit.

Carving the senescent phenotype by the chemical reactivity of catecholamines: An integrative review

Macromolecules damaged by covalent modifications produced by chemically reactive metabolites accumulate in the slowly renewable components of living bodies and compromise their functions. Among such metabolites, catecholamines (CA) are unique, compared with the ubiquitous oxygen, ROS, glucose and methylglyoxal, in that their high chemical reactivity is confined to a limited set of cell types, including the dopaminergic and noradrenergic neurons and their direct targets, which suffer from CA propensities for autoxidation yielding toxic quinones, and for Pictet-Spengler reactions with carbonyl-containing compounds, which yield mitochondrial toxins. The functions progressively compromised because of that include motor performance, cognition, reward-driven behaviors, emotional tuning, and the neuroendocrine control of reproduction. The phenotypic manifestations of the resulting disorders culminate in such conditions as Parkinson's and Alzheimer's diseases, hypertension, sarcopenia, and menopause. The reasons to suspect that CA play some special role in aging accumulated since early 1970-ies. Published reviews address the role of CA hazardousness in the development of specific aging-associated diseases. The present integrative review explores how the bizarre discrepancy between CA hazardousness and biological importance could have emerged in evolution, how much does the chemical reactivity of CA contribute to the senescent phenotype in mammals, and what can be done with it.

Context-dependent representations of movement in Drosophila dopaminergic reinforcement pathways

Dopamine plays a central role in motivating and modifying behavior, serving to invigorate current behavioral performance and guide future actions through learning. Here we examine how this single neuromodulator can contribute to such diverse forms of behavioral modulation. By recording from the dopaminergic reinforcement pathways of the Drosophila mushroom body during active odor navigation, we reveal how their ongoing motor-associated activity relates to goal-directed behavior. We found that dopaminergic neurons correlate with different behavioral variables depending on the specific navigational strategy of an animal, such that the activity of these neurons preferentially reflects the actions most relevant to odor pursuit. Furthermore, we show that these motor correlates are translated to ongoing dopamine release, and acutely perturbing dopaminergic signaling alters the strength of odor tracking. Context-dependent representations of movement and reinforcement cues are thus multiplexed within the mushroom body dopaminergic pathways, enabling them to coordinately influence both ongoing and future behavior.