1
|
Galloni AR, Yuan Y, Zhu M, Yu H, Bisht RS, Wu CTM, Grienberger C, Ramanathan S, Milstein AD. Neuromorphic one-shot learning utilizing a phase-transition material. Proc Natl Acad Sci U S A 2024; 121:e2318362121. [PMID: 38630718 PMCID: PMC11047090 DOI: 10.1073/pnas.2318362121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Accepted: 03/25/2024] [Indexed: 04/19/2024] Open
Abstract
Design of hardware based on biological principles of neuronal computation and plasticity in the brain is a leading approach to realizing energy- and sample-efficient AI and learning machines. An important factor in selection of the hardware building blocks is the identification of candidate materials with physical properties suitable to emulate the large dynamic ranges and varied timescales of neuronal signaling. Previous work has shown that the all-or-none spiking behavior of neurons can be mimicked by threshold switches utilizing material phase transitions. Here, we demonstrate that devices based on a prototypical metal-insulator-transition material, vanadium dioxide (VO2), can be dynamically controlled to access a continuum of intermediate resistance states. Furthermore, the timescale of their intrinsic relaxation can be configured to match a range of biologically relevant timescales from milliseconds to seconds. We exploit these device properties to emulate three aspects of neuronal analog computation: fast (~1 ms) spiking in a neuronal soma compartment, slow (~100 ms) spiking in a dendritic compartment, and ultraslow (~1 s) biochemical signaling involved in temporal credit assignment for a recently discovered biological mechanism of one-shot learning. Simulations show that an artificial neural network using properties of VO2 devices to control an agent navigating a spatial environment can learn an efficient path to a reward in up to fourfold fewer trials than standard methods. The phase relaxations described in our study may be engineered in a variety of materials and can be controlled by thermal, electrical, or optical stimuli, suggesting further opportunities to emulate biological learning in neuromorphic hardware.
Collapse
Affiliation(s)
- Alessandro R. Galloni
- Department of Neuroscience and Cell Biology, Robert Wood Johnson Medical School, Rutgers, The State University of New Jersey, Piscataway, NJ08854
- Center for Advanced Biotechnology and Medicine, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Yifan Yuan
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Minning Zhu
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Haoming Yu
- School of Materials Engineering, Purdue University, West Lafayette, IN47907
| | - Ravindra S. Bisht
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Chung-Tse Michael Wu
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Christine Grienberger
- Department of Neuroscience, Brandeis University, Waltham, MA02453
- Department of Biology and Volen National Center for Complex Systems, Brandeis University, Waltham, MA02453
| | - Shriram Ramanathan
- Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| | - Aaron D. Milstein
- Department of Neuroscience and Cell Biology, Robert Wood Johnson Medical School, Rutgers, The State University of New Jersey, Piscataway, NJ08854
- Center for Advanced Biotechnology and Medicine, Rutgers, The State University of New Jersey, Piscataway, NJ08854
| |
Collapse
|
2
|
Kuroki S, Mizuseki K. CA3 Circuit Model Compressing Sequential Information in Theta Oscillation and Replay. Neural Comput 2024; 36:501-548. [PMID: 38457750 DOI: 10.1162/neco_a_01641] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Accepted: 11/20/2023] [Indexed: 03/10/2024]
Abstract
The hippocampus plays a critical role in the compression and retrieval of sequential information. During wakefulness, it achieves this through theta phase precession and theta sequences. Subsequently, during periods of sleep or rest, the compressed information reactivates through sharp-wave ripple events, manifesting as memory replay. However, how these sequential neuronal activities are generated and how they store information about the external environment remain unknown. We developed a hippocampal cornu ammonis 3 (CA3) computational model based on anatomical and electrophysiological evidence from the biological CA3 circuit to address these questions. The model comprises theta rhythm inhibition, place input, and CA3-CA3 plastic recurrent connection. The model can compress the sequence of the external inputs, reproduce theta phase precession and replay, learn additional sequences, and reorganize previously learned sequences. A gradual increase in synaptic inputs, controlled by interactions between theta-paced inhibition and place inputs, explained the mechanism of sequence acquisition. This model highlights the crucial role of plasticity in the CA3 recurrent connection and theta oscillational dynamics and hypothesizes how the CA3 circuit acquires, compresses, and replays sequential information.
Collapse
Affiliation(s)
- Satoshi Kuroki
- Department of Physiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, 545-8585, Japan
| | - Kenji Mizuseki
- Department of Physiology, Graduate School of Medicine, Osaka Metropolitan University, Osaka, 545-8585, Japan
| |
Collapse
|
3
|
Sloin HE, Spivak L, Levi A, Gattegno R, Someck S, Stark E. Local activation of CA1 pyramidal cells induces theta-phase precession. Science 2024; 383:551-558. [PMID: 38301006 DOI: 10.1126/science.adk2456] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Accepted: 12/21/2023] [Indexed: 02/03/2024]
Abstract
Hippocampal theta-phase precession is involved in spatiotemporal coding and in generating multineural spike sequences, but how precession originates remains unresolved. To determine whether precession can be generated directly in hippocampal area CA1 and disambiguate multiple competing mechanisms, we used closed-loop optogenetics to impose artificial place fields in pyramidal cells of mice running on a linear track. More than one-third of the CA1 artificial fields exhibited synthetic precession that persisted for a full theta cycle. By contrast, artificial fields in the parietal cortex did not exhibit synthetic precession. These findings are incompatible with precession models based on inheritance, dual-input, spreading activation, inhibition-excitation summation, or somato-dendritic competition. Thus, a precession generator resides locally within CA1.
Collapse
Affiliation(s)
- Hadas E Sloin
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv 6997801, Israel
- Department of Physiology and Pharmacology, Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Lidor Spivak
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv 6997801, Israel
- Department of Physiology and Pharmacology, Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Amir Levi
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv 6997801, Israel
- Department of Physiology and Pharmacology, Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Roni Gattegno
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Shirly Someck
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv 6997801, Israel
- Department of Physiology and Pharmacology, Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
| | - Eran Stark
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv 6997801, Israel
- Department of Physiology and Pharmacology, Faculty of Medicine, Tel Aviv University, Tel Aviv 6997801, Israel
- Sagol Department of Neurobiology, Haifa University, Haifa 3103301, Israel
| |
Collapse
|
4
|
Yang L, Jin F, Yang L, Li J, Li Z, Li M, Shang Z. The Hippocampus in Pigeons Contributes to the Model-Based Valuation and the Relationship between Temporal Context States. Animals (Basel) 2024; 14:431. [PMID: 38338074 PMCID: PMC10854895 DOI: 10.3390/ani14030431] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2023] [Revised: 01/25/2024] [Accepted: 01/25/2024] [Indexed: 02/12/2024] Open
Abstract
Model-based decision-making guides organism behavior by the representation of the relationships between different states. Previous studies have shown that the mammalian hippocampus (Hp) plays a key role in learning the structure of relationships among experiences. However, the hippocampal neural mechanisms of birds for model-based learning have rarely been reported. Here, we trained six pigeons to perform a two-step task and explore whether their Hp contributes to model-based learning. Behavioral performance and hippocampal multi-channel local field potentials (LFPs) were recorded during the task. We estimated the subjective values using a reinforcement learning model dynamically fitted to the pigeon's choice of behavior. The results show that the model-based learner can capture the behavioral choices of pigeons well throughout the learning process. Neural analysis indicated that high-frequency (12-100 Hz) power in Hp represented the temporal context states. Moreover, dynamic correlation and decoding results provided further support for the high-frequency dependence of model-based valuations. In addition, we observed a significant increase in hippocampal neural similarity at the low-frequency band (1-12 Hz) for common temporal context states after learning. Overall, our findings suggest that pigeons use model-based inferences to learn multi-step tasks, and multiple LFP frequency bands collaboratively contribute to model-based learning. Specifically, the high-frequency (12-100 Hz) oscillations represent model-based valuations, while the low-frequency (1-12 Hz) neural similarity is influenced by the relationship between temporal context states. These results contribute to our understanding of the neural mechanisms underlying model-based learning and broaden the scope of hippocampal contributions to avian behavior.
Collapse
Affiliation(s)
- Lifang Yang
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Fuli Jin
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Long Yang
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Jiajia Li
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Zhihui Li
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
- Institute of Medical Engineering Technology and Data Mining, Zhengzhou University, Zhengzhou 450001, China
| | - Mengmeng Li
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Zhigang Shang
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
- Institute of Medical Engineering Technology and Data Mining, Zhengzhou University, Zhengzhou 450001, China
| |
Collapse
|
5
|
Huelin Gorriz M, Takigawa M, Bendor D. The role of experience in prioritizing hippocampal replay. Nat Commun 2023; 14:8157. [PMID: 38071221 PMCID: PMC10710481 DOI: 10.1038/s41467-023-43939-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Accepted: 11/24/2023] [Indexed: 12/18/2023] Open
Abstract
During sleep, recent memories are replayed by the hippocampus, leading to their consolidation, with a higher priority given to salient experiences. To examine the role of replay in the selective strengthening of memories, we recorded large ensembles of hippocampal place cells while male rats ran repeated spatial trajectories on two linear tracks, differing in either their familiarity or number of laps run. We observed that during sleep, the rate of replay events for a given track increased proportionally with the number of spatial trajectories run by the animal. In contrast, the rate of sleep replay events decreased if the animal was more familiar with the track. Furthermore, we find that the cumulative number of awake replay events occurring during behavior, influenced by both the novelty and duration of an experience, predicts which memories are prioritized for sleep replay, providing a more parsimonious neural correlate for the selective strengthening of memories.
Collapse
Affiliation(s)
- Marta Huelin Gorriz
- Institute of Behavioural Neuroscience (IBN), University College London (UCL), London, WC1H 0AP, UK
| | - Masahiro Takigawa
- Institute of Behavioural Neuroscience (IBN), University College London (UCL), London, WC1H 0AP, UK
| | - Daniel Bendor
- Institute of Behavioural Neuroscience (IBN), University College London (UCL), London, WC1H 0AP, UK.
| |
Collapse
|
6
|
Piette C, Gervasi N, Venance L. Synaptic plasticity through a naturalistic lens. Front Synaptic Neurosci 2023; 15:1250753. [PMID: 38145207 PMCID: PMC10744866 DOI: 10.3389/fnsyn.2023.1250753] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Accepted: 11/20/2023] [Indexed: 12/26/2023] Open
Abstract
From the myriad of studies on neuronal plasticity, investigating its underlying molecular mechanisms up to its behavioral relevance, a very complex landscape has emerged. Recent efforts have been achieved toward more naturalistic investigations as an attempt to better capture the synaptic plasticity underpinning of learning and memory, which has been fostered by the development of in vivo electrophysiological and imaging tools. In this review, we examine these naturalistic investigations, by devoting a first part to synaptic plasticity rules issued from naturalistic in vivo-like activity patterns. We next give an overview of the novel tools, which enable an increased spatio-temporal specificity for detecting and manipulating plasticity expressed at individual spines up to neuronal circuit level during behavior. Finally, we put particular emphasis on works considering brain-body communication loops and macroscale contributors to synaptic plasticity, such as body internal states and brain energy metabolism.
Collapse
Affiliation(s)
- Charlotte Piette
- Center for Interdisciplinary Research in Biology (CIRB), College de France, CNRS, INSERM, Université PSL, Paris, France
| | | | - Laurent Venance
- Center for Interdisciplinary Research in Biology (CIRB), College de France, CNRS, INSERM, Université PSL, Paris, France
| |
Collapse
|
7
|
Son JY, Bhandari A, FeldmanHall O. Abstract cognitive maps of social network structure aid adaptive inference. Proc Natl Acad Sci U S A 2023; 120:e2310801120. [PMID: 37963254 PMCID: PMC10666027 DOI: 10.1073/pnas.2310801120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Accepted: 10/12/2023] [Indexed: 11/16/2023] Open
Abstract
Social navigation-such as anticipating where gossip may spread, or identifying which acquaintances can help land a job-relies on knowing how people are connected within their larger social communities. Problematically, for most social networks, the space of possible relationships is too vast to observe and memorize. Indeed, people's knowledge of these social relations is well known to be biased and error-prone. Here, we reveal that these biased representations reflect a fundamental computation that abstracts over individual relationships to enable principled inferences about unseen relationships. We propose a theory of network representation that explains how people learn inferential cognitive maps of social relations from direct observation, what kinds of knowledge structures emerge as a consequence, and why it can be beneficial to encode systematic biases into social cognitive maps. Leveraging simulations, laboratory experiments, and "field data" from a real-world network, we find that people abstract observations of direct relations (e.g., friends) into inferences of multistep relations (e.g., friends-of-friends). This multistep abstraction mechanism enables people to discover and represent complex social network structure, affording adaptive inferences across a variety of contexts, including friendship, trust, and advice-giving. Moreover, this multistep abstraction mechanism unifies a variety of otherwise puzzling empirical observations about social behavior. Our proposal generalizes the theory of cognitive maps to the fundamental computational problem of social inference, presenting a powerful framework for understanding the workings of a predictive mind operating within a complex social world.
Collapse
Affiliation(s)
- Jae-Young Son
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI02912
| | - Apoorva Bhandari
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI02912
| | - Oriel FeldmanHall
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI02912
- Carney Institute for Brain Sciences, Brown University, Providence, RI02912
| |
Collapse
|
8
|
Liu C, Todorova R, Tang W, Oliva A, Fernandez-Ruiz A. Associative and predictive hippocampal codes support memory-guided behaviors. Science 2023; 382:eadi8237. [PMID: 37856604 PMCID: PMC10894649 DOI: 10.1126/science.adi8237] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Accepted: 08/21/2023] [Indexed: 10/21/2023]
Abstract
Episodic memory involves learning and recalling associations between items and their spatiotemporal context. Those memories can be further used to generate internal models of the world that enable predictions to be made. The mechanisms that support these associative and predictive aspects of memory are not yet understood. In this study, we used an optogenetic manipulation to perturb the sequential structure, but not global network dynamics, of place cells as rats traversed specific spatial trajectories. This perturbation abolished replay of those trajectories and the development of predictive representations, leading to impaired learning of new optimal trajectories during memory-guided navigation. However, place cell assembly reactivation and reward-context associative learning were unaffected. Our results show a mechanistic dissociation between two complementary hippocampal codes: an associative code (through coactivity) and a predictive code (through sequences).
Collapse
Affiliation(s)
| | | | - Wenbo Tang
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY, USA
| | - Azahara Oliva
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY, USA
| | | |
Collapse
|
9
|
Mehrotra D, Dubé L. Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus. Front Neurosci 2023; 17:1200842. [PMID: 37732307 PMCID: PMC10508350 DOI: 10.3389/fnins.2023.1200842] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 08/25/2023] [Indexed: 09/22/2023] Open
Abstract
For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.
Collapse
Affiliation(s)
- Dhruv Mehrotra
- Integrated Program in Neuroscience, McGill University, Montréal, QC, Canada
- Montréal Neurological Institute, McGill University, Montréal, QC, Canada
| | - Laurette Dubé
- Desautels Faculty of Management, McGill University, Montréal, QC, Canada
- McGill Center for the Convergence of Health and Economics, McGill University, Montréal, QC, Canada
| |
Collapse
|
10
|
Fang C, Aronov D, Abbott LF, Mackevicius EL. Neural learning rules for generating flexible predictions and computing the successor representation. eLife 2023; 12:e80680. [PMID: 36928104 PMCID: PMC10019889 DOI: 10.7554/elife.80680] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Accepted: 10/26/2022] [Indexed: 03/18/2023] Open
Abstract
The predictive nature of the hippocampus is thought to be useful for memory-guided cognitive behaviors. Inspired by the reinforcement learning literature, this notion has been formalized as a predictive map called the successor representation (SR). The SR captures a number of observations about hippocampal activity. However, the algorithm does not provide a neural mechanism for how such representations arise. Here, we show the dynamics of a recurrent neural network naturally calculate the SR when the synaptic weights match the transition probability matrix. Interestingly, the predictive horizon can be flexibly modulated simply by changing the network gain. We derive simple, biologically plausible learning rules to learn the SR in a recurrent network. We test our model with realistic inputs and match hippocampal data recorded during random foraging. Taken together, our results suggest that the SR is more accessible in neural circuits than previously thought and can support a broad range of cognitive functions.
Collapse
Affiliation(s)
- Ching Fang
- Zuckerman Institute, Department of Neuroscience, Columbia UniversityNew YorkUnited States
| | - Dmitriy Aronov
- Zuckerman Institute, Department of Neuroscience, Columbia UniversityNew YorkUnited States
| | - LF Abbott
- Zuckerman Institute, Department of Neuroscience, Columbia UniversityNew YorkUnited States
| | - Emily L Mackevicius
- Zuckerman Institute, Department of Neuroscience, Columbia UniversityNew YorkUnited States
- Basis Research InstituteNew YorkUnited States
| |
Collapse
|
11
|
Bono J, Zannone S, Pedrosa V, Clopath C. Learning predictive cognitive maps with spiking neurons during behavior and replays. eLife 2023; 12:e80671. [PMID: 36927625 PMCID: PMC10019888 DOI: 10.7554/elife.80671] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Accepted: 01/12/2023] [Indexed: 03/18/2023] Open
Abstract
The hippocampus has been proposed to encode environments using a representation that contains predictive information about likely future states, called the successor representation. However, it is not clear how such a representation could be learned in the hippocampal circuit. Here, we propose a plasticity rule that can learn this predictive map of the environment using a spiking neural network. We connect this biologically plausible plasticity rule to reinforcement learning, mathematically and numerically showing that it implements the TD-lambda algorithm. By spanning these different levels, we show how our framework naturally encompasses behavioral activity and replays, smoothly moving from rate to temporal coding, and allows learning over behavioral timescales with a plasticity rule acting on a timescale of milliseconds. We discuss how biological parameters such as dwelling times at states, neuronal firing rates and neuromodulation relate to the delay discounting parameter of the TD algorithm, and how they influence the learned representation. We also find that, in agreement with psychological studies and contrary to reinforcement learning theory, the discount factor decreases hyperbolically with time. Finally, our framework suggests a role for replays, in both aiding learning in novel environments and finding shortcut trajectories that were not experienced during behavior, in agreement with experimental data.
Collapse
Affiliation(s)
- Jacopo Bono
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| | - Sara Zannone
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| | - Victor Pedrosa
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| | - Claudia Clopath
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| |
Collapse
|