1
|
Laing PAF, Vervliet B, Dunsmoor JE, Harrison BJ. Pavlovian safety learning: An integrative theoretical review. Psychon Bull Rev 2024:10.3758/s13423-024-02559-4. [PMID: 39167292 DOI: 10.3758/s13423-024-02559-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/22/2024] [Indexed: 08/23/2024]
Abstract
Safety learning involves associating stimuli with the absence of threats, enabling the inhibition of fear and anxiety. Despite growing interest in psychology, psychiatry, and neuroscience, safety learning lacks a formal consensus definition, leading to inconsistent methodologies and varied results. Conceptualized as a form of inhibitory learning (conditioned inhibition), safety learning can be understood through formal learning theories, such as the Rescorla-Wagner and Pearce-Hall models. This review aims to establish a principled conceptualization of 'Pavlovian safety learning', identifying cognitive mechanisms that generate safety and the boundary conditions that constrain it. Based on these observations, we define Pavlovian safety learning as an active associative process, where surprising threat-omission (safety prediction error) acts as a salient reinforcing event. Instead of producing merely neutral or nonaversive states, safety learning endows stimuli with active positive associations to 'safety'. The resulting stimulus-safety memories counteract the influence of fear memories, promoting fear regulation, positive affect, and relief. We critically analyze traditional criteria of conditioned inhibition for their relevance to safety and propose areas for future innovation. A principled concept of Pavlovian safety learning may reduce methodological inconsistencies, stimulate translational research, and facilitate a comprehensive understanding of an indispensable psychological construct.
Collapse
Affiliation(s)
- Patrick A F Laing
- Department of Psychiatry and Behavioral Sciences, Dell Medical School, The University of Texas at Austin, Austin, TX, 78712, USA.
- Institute for Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA.
| | - Bram Vervliet
- Faculty of Psychology and Educational Sciences, KU Leuven, Leuven, Belgium
- Leuven Brain Institute, KU Leuven, Leuven, Belgium
| | - Joseph E Dunsmoor
- Department of Psychiatry and Behavioral Sciences, Dell Medical School, The University of Texas at Austin, Austin, TX, 78712, USA
- Institute for Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
- Department of Neuroscience, The University of Texas at Austin, Austin, TX, 78712, USA
| | - Ben J Harrison
- Melbourne Neuropsychiatry Centre, Department of Psychiatry, The University of Melbourne, Parkville, Australia
| |
Collapse
|
2
|
Dan C, Hulse BK, Kappagantula R, Jayaraman V, Hermundstad AM. A neural circuit architecture for rapid learning in goal-directed navigation. Neuron 2024; 112:2581-2599.e23. [PMID: 38795708 DOI: 10.1016/j.neuron.2024.04.036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2023] [Revised: 01/16/2024] [Accepted: 04/30/2024] [Indexed: 05/28/2024]
Abstract
Anchoring goals to spatial representations enables flexible navigation but is challenging in novel environments when both representations must be acquired simultaneously. We propose a framework for how Drosophila uses internal representations of head direction (HD) to build goal representations upon selective thermal reinforcement. We show that flies use stochastically generated fixations and directed saccades to express heading preferences in an operant visual learning paradigm and that HD neurons are required to modify these preferences based on reinforcement. We used a symmetric visual setting to expose how flies' HD and goal representations co-evolve and how the reliability of these interacting representations impacts behavior. Finally, we describe how rapid learning of new goal headings may rest on a behavioral policy whose parameters are flexible but whose form is genetically encoded in circuit architecture. Such evolutionarily structured architectures, which enable rapidly adaptive behavior driven by internal representations, may be relevant across species.
Collapse
Affiliation(s)
- Chuntao Dan
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA
| | - Brad K Hulse
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA
| | - Ramya Kappagantula
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA
| | - Vivek Jayaraman
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA.
| | - Ann M Hermundstad
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, VA 20147, USA.
| |
Collapse
|
3
|
Jesusanmi OO, Amin AA, Domcsek N, Knight JC, Philippides A, Nowotny T, Graham P. Investigating visual navigation using spiking neural network models of the insect mushroom bodies. Front Physiol 2024; 15:1379977. [PMID: 38841209 PMCID: PMC11151298 DOI: 10.3389/fphys.2024.1379977] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Accepted: 04/29/2024] [Indexed: 06/07/2024] Open
Abstract
Ants are capable of learning long visually guided foraging routes with limited neural resources. The visual scene memory needed for this behaviour is mediated by the mushroom bodies; an insect brain region important for learning and memory. In a visual navigation context, the mushroom bodies are theorised to act as familiarity detectors, guiding ants to views that are similar to those previously learned when first travelling along a foraging route. Evidence from behavioural experiments, computational studies and brain lesions all support this idea. Here we further investigate the role of mushroom bodies in visual navigation with a spiking neural network model learning complex natural scenes. By implementing these networks in GeNN-a library for building GPU accelerated spiking neural networks-we were able to test these models offline on an image database representing navigation through a complex outdoor natural environment, and also online embodied on a robot. The mushroom body model successfully learnt a large series of visual scenes (400 scenes corresponding to a 27 m route) and used these memories to choose accurate heading directions during route recapitulation in both complex environments. Through analysing our model's Kenyon cell (KC) activity, we were able to demonstrate that KC activity is directly related to the respective novelty of input images. Through conducting a parameter search we found that there is a non-linear dependence between optimal KC to visual projection neuron (VPN) connection sparsity and the length of time the model is presented with an image stimulus. The parameter search also showed training the model on lower proportions of a route generally produced better accuracy when testing on the entire route. We embodied the mushroom body model and comparator visual navigation algorithms on a Quanser Q-car robot with all processing running on an Nvidia Jetson TX2. On a 6.5 m route, the mushroom body model had a mean distance to training route (error) of 0.144 ± 0.088 m over 5 trials, which was performance comparable to standard visual-only navigation algorithms. Thus, we have demonstrated that a biologically plausible model of the ant mushroom body can navigate complex environments both in simulation and the real world. Understanding the neural basis of this behaviour will provide insight into how neural circuits are tuned to rapidly learn behaviourally relevant information from complex environments and provide inspiration for creating bio-mimetic computer/robotic systems that can learn rapidly with low energy requirements.
Collapse
Affiliation(s)
| | - Amany Azevedo Amin
- Sussex AI, School of Engineering and Informatics, University of Sussex, Brighton, United Kingdom
| | - Norbert Domcsek
- Sussex AI, School of Engineering and Informatics, University of Sussex, Brighton, United Kingdom
| | - James C. Knight
- Sussex AI, School of Engineering and Informatics, University of Sussex, Brighton, United Kingdom
| | - Andrew Philippides
- Sussex AI, School of Engineering and Informatics, University of Sussex, Brighton, United Kingdom
| | - Thomas Nowotny
- Sussex AI, School of Engineering and Informatics, University of Sussex, Brighton, United Kingdom
| | - Paul Graham
- Sussex Neuroscience, School of Life Sciences, University of Sussex, Brighton, United Kingdom
| |
Collapse
|
4
|
Webb B. Beyond prediction error: 25 years of modeling the associations formed in the insect mushroom body. Learn Mem 2024; 31:a053824. [PMID: 38862164 PMCID: PMC11199945 DOI: 10.1101/lm.053824.123] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 03/01/2024] [Indexed: 06/13/2024]
Abstract
The insect mushroom body has gained increasing attention as a system in which the computational basis of neural learning circuits can be unraveled. We now understand in detail the key locations in this circuit where synaptic associations are formed between sensory patterns and values leading to actions. However, the actual learning rule (or rules) implemented by neural activity and leading to synaptic change is still an open question. Here, I survey the diversity of answers that have been offered in computational models of this system over the past decades, including the recurring assumption-in line with top-down theories of associative learning-that the core function is to reduce prediction error. However, I will argue, a more bottom-up approach may ultimately reveal a richer algorithmic capacity in this still enigmatic brain neuropil.
Collapse
Affiliation(s)
- Barbara Webb
- School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, United Kingdom
| |
Collapse
|
5
|
Davidson AM, Hige T. Roles of feedback and feed-forward networks of dopamine subsystems: insights from Drosophila studies. Learn Mem 2024; 31:a053807. [PMID: 38862171 PMCID: PMC11199952 DOI: 10.1101/lm.053807.123] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Accepted: 11/10/2023] [Indexed: 06/13/2024]
Abstract
Across animal species, dopamine-operated memory systems comprise anatomically segregated, functionally diverse subsystems. Although individual subsystems could operate independently to support distinct types of memory, the logical interplay between subsystems is expected to enable more complex memory processing by allowing existing memory to influence future learning. Recent comprehensive ultrastructural analysis of the Drosophila mushroom body revealed intricate networks interconnecting the dopamine subsystems-the mushroom body compartments. Here, we review the functions of some of these connections that are beginning to be understood. Memory consolidation is mediated by two different forms of network: A recurrent feedback loop within a compartment maintains sustained dopamine activity required for consolidation, whereas feed-forward connections across compartments allow short-term memory formation in one compartment to open the gate for long-term memory formation in another compartment. Extinction and reversal of aversive memory rely on a similar feed-forward circuit motif that signals omission of punishment as a reward, which triggers plasticity that counteracts the original aversive memory trace. Finally, indirect feed-forward connections from a long-term memory compartment to short-term memory compartments mediate higher-order conditioning. Collectively, these emerging studies indicate that feedback control and hierarchical connectivity allow the dopamine subsystems to work cooperatively to support diverse and complex forms of learning.
Collapse
Affiliation(s)
- Andrew M Davidson
- Department of Biology, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina 27599, USA
- Department of Cell Biology and Physiology, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina 27599, USA
- Integrative Program for Biological and Genome Sciences, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina 27599, USA
| | - Toshihide Hige
- Department of Biology, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina 27599, USA
- Department of Cell Biology and Physiology, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina 27599, USA
- Integrative Program for Biological and Genome Sciences, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina 27599, USA
| |
Collapse
|
6
|
Jürgensen AM, Sakagiannis P, Schleyer M, Gerber B, Nawrot MP. Prediction error drives associative learning and conditioned behavior in a spiking model of Drosophila larva. iScience 2024; 27:108640. [PMID: 38292165 PMCID: PMC10824792 DOI: 10.1016/j.isci.2023.108640] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Revised: 11/10/2023] [Accepted: 12/01/2023] [Indexed: 02/01/2024] Open
Abstract
Predicting reinforcement from sensory cues is beneficial for goal-directed behavior. In insect brains, underlying associations between cues and reinforcement, encoded by dopaminergic neurons, are formed in the mushroom body. We propose a spiking model of the Drosophila larva mushroom body. It includes a feedback motif conveying learned reinforcement expectation to dopaminergic neurons, which can compute prediction error as the difference between expected and present reinforcement. We demonstrate that this can serve as a driving force in learning. When combined with synaptic homeostasis, our model accounts for theoretically derived features of acquisition and loss of associations that depend on the intensity of the reinforcement and its temporal proximity to the cue. From modeling olfactory learning over the time course of behavioral experiments and simulating the locomotion of individual larvae toward or away from odor sources in a virtual environment, we conclude that learning driven by prediction errors can explain larval behavior.
Collapse
Affiliation(s)
- Anna-Maria Jürgensen
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, 50674 Cologne, Germany
| | - Panagiotis Sakagiannis
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, 50674 Cologne, Germany
| | - Michael Schleyer
- Leibniz Institute for Neurobiology (LIN), Department of Genetics, 39118 Magdeburg, Germany
- Institute for the Advancement of Higher Education, Faculty of Science, Hokkaido University, Sapporo 060-08080, Japan
| | - Bertram Gerber
- Leibniz Institute for Neurobiology (LIN), Department of Genetics, 39118 Magdeburg, Germany
- Institute for Biology, Otto-von-Guericke University, 39120 Magdeburg, Germany
- Center for Brain and Behavioral Sciences (CBBS), Otto-von-Guericke University, 39118 Magdeburg, Germany
| | - Martin Paul Nawrot
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, 50674 Cologne, Germany
| |
Collapse
|
7
|
Jürgensen AM, Schmitt FJ, Nawrot MP. Minimal circuit motifs for second-order conditioning in the insect mushroom body. Front Physiol 2024; 14:1326307. [PMID: 38269060 PMCID: PMC10806035 DOI: 10.3389/fphys.2023.1326307] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2023] [Accepted: 12/22/2023] [Indexed: 01/26/2024] Open
Abstract
In well-established first-order conditioning experiments, the concurrence of a sensory cue with reinforcement forms an association, allowing the cue to predict future reinforcement. In the insect mushroom body, a brain region central to learning and memory, such associations are encoded in the synapses between its intrinsic and output neurons. This process is mediated by the activity of dopaminergic neurons that encode reinforcement signals. In second-order conditioning, a new sensory cue is paired with an already established one that presumably activates dopaminergic neurons due to its predictive power of the reinforcement. We explored minimal circuit motifs in the mushroom body for their ability to support second-order conditioning using mechanistic models. We found that dopaminergic neurons can either be activated directly by the mushroom body's intrinsic neurons or via feedback from the output neurons via several pathways. We demonstrated that the circuit motifs differ in their computational efficiency and robustness. Beyond previous research, we suggest an additional motif that relies on feedforward input of the mushroom body intrinsic neurons to dopaminergic neurons as a promising candidate for experimental evaluation. It differentiates well between trained and novel stimuli, demonstrating robust performance across a range of model parameters.
Collapse
Affiliation(s)
- Anna-Maria Jürgensen
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Cologne, Germany
| | | | | |
Collapse
|
8
|
Arican C, Schmitt FJ, Rössler W, Strube-Bloss MF, Nawrot MP. The mushroom body output encodes behavioral decision during sensory-motor transformation. Curr Biol 2023; 33:4217-4224.e4. [PMID: 37657449 DOI: 10.1016/j.cub.2023.08.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 06/30/2023] [Accepted: 08/03/2023] [Indexed: 09/03/2023]
Abstract
Animals form a behavioral decision by evaluating sensory evidence on the background of past experiences and the momentary motivational state. In insects, we still lack understanding of how and at which stage of the recurrent sensory-motor pathway behavioral decisions are formed. The mushroom body (MB), a central brain structure in insects1 and crustaceans,2,3 integrates sensory input of different modalities4,5,6 with the internal state, the behavioral state, and external sensory context7,8,9,10 through a large number of recurrent, mostly neuromodulatory inputs,11,12 implicating a functional role for MBs in state-dependent sensory-motor transformation.13,14 A number of classical conditioning studies in honeybees15,16 and fruit flies17,18,19 have provided accumulated evidence that at its output, the MB encodes the valence of a sensory stimulus with respect to its behavioral relevance. Recent work has extended this notion of valence encoding to the context of innate behaviors.8,20,21,22 Here, we co-analyzed a defined feeding behavior and simultaneous extracellular single-unit recordings from MB output neurons (MBONs) in the cockroach in response to timed sensory stimulation with odors. We show that clear neuronal responses occurred almost exclusively during behaviorally responded trials. Early MBON responses to the sensory stimulus preceded the feeding behavior and predicted its occurrence or non-occurrence from the single-trial population activity. Our results therefore suggest that at its output, the MB does not merely encode sensory stimulus valence. We hypothesize instead that the MB output represents an integrated signal of internal state, momentary environmental conditions, and experience-dependent memory to encode a behavioral decision.
Collapse
Affiliation(s)
- Cansu Arican
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Zülpicher Str. 47b, 50674 Cologne, Germany.
| | - Felix Johannes Schmitt
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Zülpicher Str. 47b, 50674 Cologne, Germany
| | - Wolfgang Rössler
- Behavioral Physiology and Sociobiology (Zoology II), Biozentrum, University of Würzburg, Am Hubland, 97074 Würzburg, Germany
| | - Martin Fritz Strube-Bloss
- Department of Biological Cybernetics and Theoretical Biology, University of Bielefeld, Universitätsstr. 25, 33615 Bielefeld, Germany
| | - Martin Paul Nawrot
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Zülpicher Str. 47b, 50674 Cologne, Germany.
| |
Collapse
|
9
|
Rajagopalan AE, Darshan R, Hibbard KL, Fitzgerald JE, Turner GC. Reward expectations direct learning and drive operant matching in Drosophila. Proc Natl Acad Sci U S A 2023; 120:e2221415120. [PMID: 37733736 PMCID: PMC10523640 DOI: 10.1073/pnas.2221415120] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2023] [Accepted: 08/11/2023] [Indexed: 09/23/2023] Open
Abstract
Foraging animals must use decision-making strategies that dynamically adapt to the changing availability of rewards in the environment. A wide diversity of animals do this by distributing their choices in proportion to the rewards received from each option, Herrnstein's operant matching law. Theoretical work suggests an elegant mechanistic explanation for this ubiquitous behavior, as operant matching follows automatically from simple synaptic plasticity rules acting within behaviorally relevant neural circuits. However, no past work has mapped operant matching onto plasticity mechanisms in the brain, leaving the biological relevance of the theory unclear. Here, we discovered operant matching in Drosophila and showed that it requires synaptic plasticity that acts in the mushroom body and incorporates the expectation of reward. We began by developing a dynamic foraging paradigm to measure choices from individual flies as they learn to associate odor cues with probabilistic rewards. We then built a model of the fly mushroom body to explain each fly's sequential choice behavior using a family of biologically realistic synaptic plasticity rules. As predicted by past theoretical work, we found that synaptic plasticity rules could explain fly matching behavior by incorporating stimulus expectations, reward expectations, or both. However, by optogenetically bypassing the representation of reward expectation, we abolished matching behavior and showed that the plasticity rule must specifically incorporate reward expectations. Altogether, these results reveal the first synapse-level mechanisms of operant matching and provide compelling evidence for the role of reward expectation signals in the fly brain.
Collapse
Affiliation(s)
- Adithya E. Rajagopalan
- Janelia Research Campus, HHMI, Ashburn, VA20147
- Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Baltimore, MD21205
| | - Ran Darshan
- Janelia Research Campus, HHMI, Ashburn, VA20147
- Department of Physiology and Pharmacology, Sackler Faculty of Medicine, Sagol School of Neuroscience, The School of Physics and Astronomy, Tel Aviv University, Tel Aviv6997801, Israel
| | | | | | | |
Collapse
|
10
|
Perisse E, Miranda M, Trouche S. Modulation of aversive value coding in the vertebrate and invertebrate brain. Curr Opin Neurobiol 2023; 79:102696. [PMID: 36871400 DOI: 10.1016/j.conb.2023.102696] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2022] [Revised: 01/30/2023] [Accepted: 01/31/2023] [Indexed: 03/06/2023]
Abstract
Avoiding potentially dangerous situations is key for the survival of any organism. Throughout life, animals learn to avoid environments, stimuli or actions that can lead to bodily harm. While the neural bases for appetitive learning, evaluation and value-based decision-making have received much attention, recent studies have revealed more complex computations for aversive signals during learning and decision-making than previously thought. Furthermore, previous experience, internal state and systems level appetitive-aversive interactions seem crucial for learning specific aversive value signals and making appropriate choices. The emergence of novel methodologies (computation analysis coupled with large-scale neuronal recordings, neuronal manipulations at unprecedented resolution offered by genetics, viral strategies and connectomics) has helped to provide novel circuit-based models for aversive (and appetitive) valuation. In this review, we focus on recent vertebrate and invertebrate studies yielding strong evidence that aversive value information can be computed by a multitude of interacting brain regions, and that past experience can modulate future aversive learning and therefore influence value-based decisions.
Collapse
Affiliation(s)
- Emmanuel Perisse
- Institute of Functional Genomics, University of Montpellier, CNRS, Inserm, 141 rue de la Cardonille, 34094 Montpellier Cedex 5, France.
| | - Magdalena Miranda
- Institute of Functional Genomics, University of Montpellier, CNRS, Inserm, 141 rue de la Cardonille, 34094 Montpellier Cedex 5, France
| | - Stéphanie Trouche
- Institute of Functional Genomics, University of Montpellier, CNRS, Inserm, 141 rue de la Cardonille, 34094 Montpellier Cedex 5, France.
| |
Collapse
|
11
|
Lipshutz D, Kashalikar A, Farashahi S, Chklovskii DB. A linear discriminant analysis model of imbalanced associative learning in the mushroom body compartment. PLoS Comput Biol 2023; 19:e1010864. [PMID: 36745688 PMCID: PMC9934445 DOI: 10.1371/journal.pcbi.1010864] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Revised: 02/16/2023] [Accepted: 01/10/2023] [Indexed: 02/07/2023] Open
Abstract
To adapt to their environments, animals learn associations between sensory stimuli and unconditioned stimuli. In invertebrates, olfactory associative learning primarily occurs in the mushroom body, which is segregated into separate compartments. Within each compartment, Kenyon cells (KCs) encoding sparse odor representations project onto mushroom body output neurons (MBONs) whose outputs guide behavior. Associated with each compartment is a dopamine neuron (DAN) that modulates plasticity of the KC-MBON synapses within the compartment. Interestingly, DAN-induced plasticity of the KC-MBON synapse is imbalanced in the sense that it only weakens the synapse and is temporally sparse. We propose a normative mechanistic model of the MBON as a linear discriminant analysis (LDA) classifier that predicts the presence of an unconditioned stimulus (class identity) given a KC odor representation (feature vector). Starting from a principled LDA objective function and under the assumption of temporally sparse DAN activity, we derive an online algorithm which maps onto the mushroom body compartment. Our model accounts for the imbalanced learning at the KC-MBON synapse and makes testable predictions that provide clear contrasts with existing models.
Collapse
Affiliation(s)
- David Lipshutz
- Center for Computational Neuroscience, Flatiron Institute, New York, New York, United States of America
- * E-mail:
| | - Aneesh Kashalikar
- Center for Computational Neuroscience, Flatiron Institute, New York, New York, United States of America
| | - Shiva Farashahi
- Center for Computational Neuroscience, Flatiron Institute, New York, New York, United States of America
| | - Dmitri B. Chklovskii
- Center for Computational Neuroscience, Flatiron Institute, New York, New York, United States of America
- Neuroscience Institute, New York University School of Medicine, New York, New York, United States of America
| |
Collapse
|
12
|
Vijayan V, Wang Z, Chandra V, Chakravorty A, Li R, Sarbanes SL, Akhlaghpour H, Maimon G. An internal expectation guides Drosophila egg-laying decisions. SCIENCE ADVANCES 2022; 8:eabn3852. [PMID: 36306348 PMCID: PMC9616500 DOI: 10.1126/sciadv.abn3852] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 05/10/2023]
Abstract
To better understand how animals make ethologically relevant decisions, we studied egg-laying substrate choice in Drosophila. We found that flies dynamically increase or decrease their egg-laying rates while exploring substrates so as to target eggs to the best, recently visited option. Visiting the best option typically yielded inhibition of egg laying on other substrates for many minutes. Our data support a model in which flies compare the current substrate's value with an internally constructed expectation on the value of available options to regulate the likelihood of laying an egg. We show that dopamine neuron activity is critical for learning and/or expressing this expectation, similar to its role in certain tasks in vertebrates. Integrating sensory experiences over minutes to generate an estimate of the quality of available options allows flies to use a dynamic reference point for judging the current substrate and might be a general way in which decisions are made.
Collapse
|
13
|
Villar ME, Pavão-Delgado M, Amigo M, Jacob PF, Merabet N, Pinot A, Perry SA, Waddell S, Perisse E. Differential coding of absolute and relative aversive value in the Drosophila brain. Curr Biol 2022; 32:4576-4592.e5. [DOI: 10.1016/j.cub.2022.08.058] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Revised: 06/24/2022] [Accepted: 08/19/2022] [Indexed: 11/30/2022]
|
14
|
Hancock CE, Rostami V, Rachad EY, Deimel SH, Nawrot MP, Fiala A. Visualization of learning-induced synaptic plasticity in output neurons of the Drosophila mushroom body γ-lobe. Sci Rep 2022; 12:10421. [PMID: 35729203 PMCID: PMC9213513 DOI: 10.1038/s41598-022-14413-5] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2022] [Accepted: 06/07/2022] [Indexed: 12/21/2022] Open
Abstract
By learning, through experience, which stimuli coincide with dangers, it is possible to predict outcomes and act pre-emptively to ensure survival. In insects, this process is localized to the mushroom body (MB), the circuitry of which facilitates the coincident detection of sensory stimuli and punishing or rewarding cues and, downstream, the execution of appropriate learned behaviors. Here, we focused our attention on the mushroom body output neurons (MBONs) of the γ-lobes that act as downstream synaptic partners of the MB γ-Kenyon cells (KCs) to ask how the output of the MB γ-lobe is shaped by olfactory associative conditioning, distinguishing this from non-associative stimulus exposure effects, and without the influence of downstream modulation. This was achieved by employing a subcellularly localized calcium sensor to specifically monitor activity at MBON postsynaptic sites. Therein, we identified a robust associative modulation within only one MBON postsynaptic compartment (MBON-γ1pedc > α/β), which displayed a suppressed postsynaptic response to an aversively paired odor. While this MBON did not undergo non-associative modulation, the reverse was true across the remainder of the γ-lobe, where general odor-evoked adaptation was observed, but no conditioned odor-specific modulation. In conclusion, associative synaptic plasticity underlying aversive olfactory learning is localized to one distinct synaptic γKC-to-γMBON connection.
Collapse
Affiliation(s)
- Clare E Hancock
- Molecular Neurobiology of Behavior, Johann-Friedrich-Blumenbach-Institute of Zoology and Anthropology, University of Göttingen, Julia-Lermontowa-Weg 3, 37077, Göttingen, Germany
| | - Vahid Rostami
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Zülpicherstraße 47b, 50674, Cologne, Germany
| | - El Yazid Rachad
- Molecular Neurobiology of Behavior, Johann-Friedrich-Blumenbach-Institute of Zoology and Anthropology, University of Göttingen, Julia-Lermontowa-Weg 3, 37077, Göttingen, Germany
| | - Stephan H Deimel
- Molecular Neurobiology of Behavior, Johann-Friedrich-Blumenbach-Institute of Zoology and Anthropology, University of Göttingen, Julia-Lermontowa-Weg 3, 37077, Göttingen, Germany
| | - Martin P Nawrot
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Zülpicherstraße 47b, 50674, Cologne, Germany
| | - André Fiala
- Molecular Neurobiology of Behavior, Johann-Friedrich-Blumenbach-Institute of Zoology and Anthropology, University of Göttingen, Julia-Lermontowa-Weg 3, 37077, Göttingen, Germany.
| |
Collapse
|
15
|
Abstract
Autonomous robots are expected to perform a wide range of sophisticated tasks in complex, unknown environments. However, available onboard computing capabilities and algorithms represent a considerable obstacle to reaching higher levels of autonomy, especially as robots get smaller and the end of Moore's law approaches. Here, we argue that inspiration from insect intelligence is a promising alternative to classic methods in robotics for the artificial intelligence (AI) needed for the autonomy of small, mobile robots. The advantage of insect intelligence stems from its resource efficiency (or parsimony) especially in terms of power and mass. First, we discuss the main aspects of insect intelligence underlying this parsimony: embodiment, sensory-motor coordination, and swarming. Then, we take stock of where insect-inspired AI stands as an alternative to other approaches to important robotic tasks such as navigation and identify open challenges on the road to its more widespread adoption. Last, we reflect on the types of processors that are suitable for implementing insect-inspired AI, from more traditional ones such as microcontrollers and field-programmable gate arrays to unconventional neuromorphic processors. We argue that even for neuromorphic processors, one should not simply apply existing AI algorithms but exploit insights from natural insect intelligence to get maximally efficient AI for robot autonomy.
Collapse
Affiliation(s)
- G C H E de Croon
- Micro Air Vehicle Laboratory, Faculty of Aerospace Engineering, TU Delft, Delft, Netherlands
| | - J J G Dupeyroux
- Micro Air Vehicle Laboratory, Faculty of Aerospace Engineering, TU Delft, Delft, Netherlands
| | - S B Fuller
- Autonomous Insect Robotics Laboratory, Department of Mechanical Engineering and Paul G. Allen School of Computer Science, University of Washington, Seattle, WA, USA
| | - J A R Marshall
- Opteran Technologies, Sheffield, UK
- Complex Systems Modeling Group, Department of Computer Science, University of Sheffield, Sheffield, UK
| |
Collapse
|
16
|
Gkanias E, McCurdy LY, Nitabach MN, Webb B. An incentive circuit for memory dynamics in the mushroom body of Drosophila melanogaster. eLife 2022; 11:75611. [PMID: 35363138 PMCID: PMC8975552 DOI: 10.7554/elife.75611] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2021] [Accepted: 03/07/2022] [Indexed: 11/30/2022] Open
Abstract
Insects adapt their response to stimuli, such as odours, according to their pairing with positive or negative reinforcements, such as sugar or shock. Recent electrophysiological and imaging findings in Drosophila melanogaster allow detailed examination of the neural mechanisms supporting the acquisition, forgetting, and assimilation of memories. We propose that this data can be explained by the combination of a dopaminergic plasticity rule that supports a variety of synaptic strength change phenomena, and a circuit structure (derived from neuroanatomy) between dopaminergic and output neurons that creates different roles for specific neurons. Computational modelling shows that this circuit allows for rapid memory acquisition, transfer from short term to long term, and exploration/exploitation trade-off. The model can reproduce the observed changes in the activity of each of the identified neurons in conditioning paradigms and can be used for flexible behavioural control.
Collapse
Affiliation(s)
- Evripidis Gkanias
- Institute of Perception Action and Behaviour, School of Informatics, University of Edinburgh, Edinburgh, United Kingdom
| | - Li Yan McCurdy
- Department of Cellular and Molecular Physiology, Yale University, New Haven, United States
| | - Michael N Nitabach
- Department of Cellular and Molecular Physiology, Yale University, New Haven, United States.,Department of Genetics, Yale University, New Haven, United States.,Department of Neuroscience, Yale University, New Haven, United States
| | - Barbara Webb
- Institute of Perception Action and Behaviour, School of Informatics, University of Edinburgh, Edinburgh, United Kingdom
| |
Collapse
|
17
|
Stahl A, Noyes NC, Boto T, Botero V, Broyles CN, Jing M, Zeng J, King LB, Li Y, Davis RL, Tomchik SM. Associative learning drives longitudinally graded presynaptic plasticity of neurotransmitter release along axonal compartments. eLife 2022; 11:76712. [PMID: 35285796 PMCID: PMC8956283 DOI: 10.7554/elife.76712] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2021] [Accepted: 03/11/2022] [Indexed: 12/27/2022] Open
Abstract
Anatomical and physiological compartmentalization of neurons is a mechanism to increase the computational capacity of a circuit, and a major question is what role axonal compartmentalization plays. Axonal compartmentalization may enable localized, presynaptic plasticity to alter neuronal output in a flexible, experience-dependent manner. Here, we show that olfactory learning generates compartmentalized, bidirectional plasticity of acetylcholine release that varies across the longitudinal compartments of Drosophila mushroom body (MB) axons. The directionality of the learning-induced plasticity depends on the valence of the learning event (aversive vs. appetitive), varies linearly across proximal to distal compartments following appetitive conditioning, and correlates with learning-induced changes in downstream mushroom body output neurons (MBONs) that modulate behavioral action selection. Potentiation of acetylcholine release was dependent on the CaV2.1 calcium channel subunit cacophony. In addition, contrast between the positive conditioned stimulus and other odors required the inositol triphosphate receptor, which maintained responsivity to odors upon repeated presentations, preventing adaptation. Downstream from the MB, a set of MBONs that receive their input from the γ3 MB compartment were required for normal appetitive learning, suggesting that they represent a key node through which reward learning influences decision-making. These data demonstrate that learning drives valence-correlated, compartmentalized, bidirectional potentiation, and depression of synaptic neurotransmitter release, which rely on distinct mechanisms and are distributed across axonal compartments in a learning circuit.
Collapse
Affiliation(s)
- Aaron Stahl
- Department of Neuroscience, The Scripps Research Institute, Jupiter, United States
| | - Nathaniel C Noyes
- Department of Neuroscience, The Scripps Research Institute, Jupiter, United States
| | - Tamara Boto
- Department of Neuroscience, The Scripps Research Institute, Jupiter, United States
| | - Valentina Botero
- Department of Neuroscience, The Scripps Research Institute, Jupiter, United States
| | - Connor N Broyles
- Department of Neuroscience, The Scripps Research Institute, Jupiter, United States
| | - Miao Jing
- Chinese Institute for Brain Research, Beijing, China
| | - Jianzhi Zeng
- Peking-Tsinghua Center for Life Sciences, Peking University, Beijing, China.,State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China.,PKU IDG/McGovern Institute for Brain Research, Beijing, China
| | - Lanikea B King
- Department of Neuroscience, The Scripps Research Institute, Jupiter, United States
| | - Yulong Li
- Chinese Institute for Brain Research, Beijing, China.,Peking-Tsinghua Center for Life Sciences, Peking University, Beijing, China.,State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China.,PKU IDG/McGovern Institute for Brain Research, Beijing, China
| | - Ronald L Davis
- Department of Neuroscience, The Scripps Research Institute, Jupiter, United States
| | - Seth M Tomchik
- Department of Neuroscience, The Scripps Research Institute, Jupiter, United States
| |
Collapse
|
18
|
Bennett JEM, Philippides A, Nowotny T. Learning with reinforcement prediction errors in a model of the Drosophila mushroom body. Nat Commun 2021; 12:2569. [PMID: 33963189 PMCID: PMC8105414 DOI: 10.1038/s41467-021-22592-4] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2019] [Accepted: 03/16/2021] [Indexed: 02/03/2023] Open
Abstract
Effective decision making in a changing environment demands that accurate predictions are learned about decision outcomes. In Drosophila, such learning is orchestrated in part by the mushroom body, where dopamine neurons signal reinforcing stimuli to modulate plasticity presynaptic to mushroom body output neurons. Building on previous mushroom body models, in which dopamine neurons signal absolute reinforcement, we propose instead that dopamine neurons signal reinforcement prediction errors by utilising feedback reinforcement predictions from output neurons. We formulate plasticity rules that minimise prediction errors, verify that output neurons learn accurate reinforcement predictions in simulations, and postulate connectivity that explains more physiological observations than an experimentally constrained model. The constrained and augmented models reproduce a broad range of conditioning and blocking experiments, and we demonstrate that the absence of blocking does not imply the absence of prediction error dependent learning. Our results provide five predictions that can be tested using established experimental methods.
Collapse
Affiliation(s)
- James E. M. Bennett
- grid.12082.390000 0004 1936 7590Department of Informatics, University of Sussex, Brighton, UK
| | - Andrew Philippides
- grid.12082.390000 0004 1936 7590Department of Informatics, University of Sussex, Brighton, UK
| | - Thomas Nowotny
- grid.12082.390000 0004 1936 7590Department of Informatics, University of Sussex, Brighton, UK
| |
Collapse
|
19
|
Springer M, Nawrot MP. A Mechanistic Model for Reward Prediction and Extinction Learning in the Fruit Fly. eNeuro 2021; 8:ENEURO.0549-20.2021. [PMID: 33785523 PMCID: PMC8211469 DOI: 10.1523/eneuro.0549-20.2021] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2020] [Revised: 03/15/2021] [Accepted: 03/18/2021] [Indexed: 01/08/2023] Open
Abstract
Extinction learning, the ability to update previously learned information by integrating novel contradictory information, is of high clinical relevance for therapeutic approaches to the modulation of maladaptive memories. Insect models have been instrumental in uncovering fundamental processes of memory formation and memory update. Recent experimental results in Drosophila melanogaster suggest that, after the behavioral extinction of a memory, two parallel but opposing memory traces coexist, residing at different sites within the mushroom body (MB). Here, we propose a minimalistic circuit model of the Drosophila MB that supports classical appetitive and aversive conditioning and memory extinction. The model is tailored to the existing anatomic data and involves two circuit motives of central functional importance. It employs plastic synaptic connections between Kenyon cells (KCs) and MB output neurons (MBONs) in separate and mutually inhibiting appetitive and aversive learning pathways. Recurrent modulation of plasticity through projections from MBONs to reinforcement-mediating dopaminergic neurons (DAN) implements a simple reward prediction mechanism. A distinct set of four MBONs encodes odor valence and predicts behavioral model output. Subjecting our model to learning and extinction protocols reproduced experimental results from recent behavioral and imaging studies. Simulating the experimental blocking of synaptic output of individual neurons or neuron groups in the model circuit confirmed experimental results and allowed formulation of testable predictions. In the temporal domain, our model achieves rapid learning with a step-like increase in the encoded odor value after a single pairing of the conditioned stimulus (CS) with a reward or punishment, facilitating single-trial learning.
Collapse
Affiliation(s)
- Magdalena Springer
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Biocenter, Cologne 50674, Germany
| | - Martin Paul Nawrot
- Computational Systems Neuroscience, Institute of Zoology, University of Cologne, Biocenter, Cologne 50674, Germany
| |
Collapse
|