1
|
Wei H, Jin X, Su Z. A Circuit Model for Working Memory Based on Hybrid Positive and Negative-Derivative Feedback Mechanism. Brain Sci 2022; 12:547. [PMID: 35624934 PMCID: PMC9139460 DOI: 10.3390/brainsci12050547] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2022] [Revised: 03/28/2022] [Accepted: 04/22/2022] [Indexed: 12/10/2022] Open
Abstract
Working memory (WM) plays an important role in cognitive activity. The WM system is used to temporarily store information in learning and decision-making. WM always functions in many aspects of daily life, such as the short-term memory of words, cell phone verification codes, and cell phone numbers. In young adults, studies have shown that a central memory store is limited to three to five meaningful items. Little is known about how WM functions at the microscopic neural level, but appropriate neural network computational models can help us gain a better understanding of it. In this study, we attempt to design a microscopic neural network model to explain the internal mechanism of WM. The performance of existing positive feedback models depends on the parameters of a synapse. We use a negative-derivative feedback mechanism to counteract the drift in persistent activity, making the hybrid positive and negative-derivative feedback (HPNF) model more robust to common disturbances. To fulfill the mechanism of WM at the neural circuit level, we construct two main neural networks based on the HPNF model: a memory-storage sub-network (the memory-storage sub-network is composed of several sets of neurons, so we call it "SET network", or "SET" for short) with positive feedback and negative-derivative feedback and a storage distribution network (SDN) designed by combining SET for memory item storage and memory updating. The SET network is a neural information self-sustaining mechanism, which is robust to common disturbances; the SDN constructs a storage distribution network at the neural circuit level; the experimental results show that our network can fulfill the storage, association, updating, and forgetting of information at the level of neural circuits, and it can work in different individuals with little change in parameters.
Collapse
Affiliation(s)
- Hui Wei
- Laboratory of Cognitive Model and Algorithm, Department of Computer Science, Fudan University, No. 825 Zhangheng Road, Shanghai 201203, China; (X.J.); (Z.S.)
- Shanghai Key Laboratory of Data Science, No. 220 Handan Road, Shanghai 200433, China
| | - Xiao Jin
- Laboratory of Cognitive Model and Algorithm, Department of Computer Science, Fudan University, No. 825 Zhangheng Road, Shanghai 201203, China; (X.J.); (Z.S.)
- Shanghai Key Laboratory of Data Science, No. 220 Handan Road, Shanghai 200433, China
| | - Zihao Su
- Laboratory of Cognitive Model and Algorithm, Department of Computer Science, Fudan University, No. 825 Zhangheng Road, Shanghai 201203, China; (X.J.); (Z.S.)
- Shanghai Key Laboratory of Data Science, No. 220 Handan Road, Shanghai 200433, China
| |
Collapse
|
2
|
Prediction-error neurons in circuits with multiple neuron types: Formation, refinement, and functional implications. Proc Natl Acad Sci U S A 2022; 119:e2115699119. [PMID: 35320037 PMCID: PMC9060484 DOI: 10.1073/pnas.2115699119] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/14/2023] Open
Abstract
An influential idea in neuroscience is that neural circuits do not only passively process sensory information but rather actively compare them with predictions thereof. A core element of this comparison is prediction-error neurons, the activity of which only changes upon mismatches between actual and predicted sensory stimuli. While it has been shown that these prediction-error neurons come in different variants, it is largely unresolved how they are simultaneously formed and shaped by highly interconnected neural networks. By using a computational model, we study the circuit-level mechanisms that give rise to different variants of prediction-error neurons. Our results shed light on the formation, refinement, and robustness of prediction-error circuits, an important step toward a better understanding of predictive processing. Predictable sensory stimuli do not evoke significant responses in a subset of cortical excitatory neurons. Some of those neurons, however, change their activity upon mismatches between actual and predicted stimuli. Different variants of these prediction-error neurons exist, and they differ in their responses to unexpected sensory stimuli. However, it is unclear how these variants can develop and coexist in the same recurrent network and how they are simultaneously shaped by the astonishing diversity of inhibitory interneurons. Here, we study these questions in a computational network model with three types of inhibitory interneurons. We find that balancing excitation and inhibition in multiple pathways gives rise to heterogeneous prediction-error circuits. Dependent on the network’s initial connectivity and distribution of actual and predicted sensory inputs, these circuits can form different variants of prediction-error neurons that are robust to network perturbations and generalize to stimuli not seen during learning. These variants can be learned simultaneously via homeostatic inhibitory plasticity with low baseline firing rates. Finally, we demonstrate that prediction-error neurons can support biased perception, we illustrate a number of functional implications, and we discuss testable predictions.
Collapse
|
3
|
Yoo AH, Collins AGE. How Working Memory and Reinforcement Learning Are Intertwined: A Cognitive, Neural, and Computational Perspective. J Cogn Neurosci 2021; 34:551-568. [PMID: 34942642 DOI: 10.1162/jocn_a_01808] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Reinforcement learning and working memory are two core processes of human cognition and are often considered cognitively, neuroscientifically, and algorithmically distinct. Here, we show that the brain networks that support them actually overlap significantly and that they are less distinct cognitive processes than often assumed. We review literature demonstrating the benefits of considering each process to explain properties of the other and highlight recent work investigating their more complex interactions. We discuss how future research in both computational and cognitive sciences can benefit from one another, suggesting that a key missing piece for artificial agents to learn to behave with more human-like efficiency is taking working memory's role in learning seriously. This review highlights the risks of neglecting the interplay between different processes when studying human behavior (in particular when considering individual differences). We emphasize the importance of investigating these dynamics to build a comprehensive understanding of human cognition.
Collapse
|
4
|
Womelsdorf T, Watson MR, Tiesinga P. Learning at Variable Attentional Load Requires Cooperation of Working Memory, Meta-learning, and Attention-augmented Reinforcement Learning. J Cogn Neurosci 2021; 34:79-107. [PMID: 34813644 PMCID: PMC9830786 DOI: 10.1162/jocn_a_01780] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/12/2023]
Abstract
Flexible learning of changing reward contingencies can be realized with different strategies. A fast learning strategy involves using working memory of recently rewarded objects to guide choices. A slower learning strategy uses prediction errors to gradually update value expectations to improve choices. How the fast and slow strategies work together in scenarios with real-world stimulus complexity is not well known. Here, we aim to disentangle their relative contributions in rhesus monkeys while they learned the relevance of object features at variable attentional load. We found that learning behavior across six monkeys is consistently best predicted with a model combining (i) fast working memory and (ii) slower reinforcement learning from differently weighted positive and negative prediction errors as well as (iii) selective suppression of nonchosen feature values and (iv) a meta-learning mechanism that enhances exploration rates based on a memory trace of recent errors. The optimal model parameter settings suggest that these mechanisms cooperate differently at low and high attentional loads. Whereas working memory was essential for efficient learning at lower attentional loads, enhanced weighting of negative prediction errors and meta-learning were essential for efficient learning at higher attentional loads. Together, these findings pinpoint a canonical set of learning mechanisms and suggest how they may cooperate when subjects flexibly adjust to environments with variable real-world attentional demands.
Collapse
Affiliation(s)
- Thilo Womelsdorf
- Department of Psychology, Vanderbilt University, Nashville, TN 37240
| | - Marcus R. Watson
- School of Kinesiology and Health Science, Centre for Vision Research, York University, 4700 Keele Street, Toronto, Ontario M6J 1P3, Canada
| | - Paul Tiesinga
- Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen 6525 EN, Netherlands
| |
Collapse
|
5
|
Zambrano D, Roelfsema PR, Bohte S. Learning continuous-time working memory tasks with on-policy neural reinforcement learning. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.11.072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
6
|
Circuit mechanisms for cortical plasticity and learning. Semin Cell Dev Biol 2021; 125:68-75. [PMID: 34332885 DOI: 10.1016/j.semcdb.2021.07.012] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2021] [Revised: 07/15/2021] [Accepted: 07/16/2021] [Indexed: 11/22/2022]
Abstract
The cerebral cortex integrates sensory information with emotional states and internal representations to produce coherent percepts, form associations, and execute voluntary actions. For the cortex to optimize perception, its neuronal network needs to dynamically retrieve and encode new information. Over the last few decades, research has started to provide insight into how the cortex serves these functions. Building on classical Hebbian plasticity models, the latest hypotheses hold that throughout experience and learning, streams of feedforward, feedback, and modulatory information operate in selective and coordinated manners to alter the strength of synapses and ultimately change the response properties of cortical neurons. Here, we describe cortical plasticity mechanisms that involve the concerted action of feedforward and long-range feedback input onto pyramidal neurons as well as the implication of local disinhibitory circuit motifs in this process.
Collapse
|
7
|
Lin X, Zou X, Ji Z, Huang T, Wu S, Mi Y. A brain-inspired computational model for spatio-temporal information processing. Neural Netw 2021; 143:74-87. [PMID: 34091238 DOI: 10.1016/j.neunet.2021.05.015] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2020] [Revised: 03/31/2021] [Accepted: 05/12/2021] [Indexed: 11/28/2022]
Abstract
Spatio-temporal information processing is fundamental in both brain functions and AI applications. Current strategies for spatio-temporal pattern recognition usually involve explicit feature extraction followed by feature aggregation, which requires a large amount of labeled data. In the present study, motivated by the subcortical visual pathway and early stages of the auditory pathway for motion and sound processing, we propose a novel brain-inspired computational model for generic spatio-temporal pattern recognition. The model consists of two modules, a reservoir module and a decision-making module. The former projects complex spatio-temporal patterns into spatially separated neural representations via its recurrent dynamics, the latter reads out neural representations via integrating information over time, and the two modules are linked together using known examples. Using synthetic data, we demonstrate that the model can extract the frequency and order information of temporal inputs. We apply the model to reproduce the looming pattern discrimination behavior as observed in experiments successfully. Furthermore, we apply the model to the gait recognition task, and demonstrate that our model accomplishes the recognition in an event-based manner and outperforms deep learning counterparts when training data is limited.
Collapse
Affiliation(s)
- Xiaohan Lin
- School of Electronics Engineering and Computer Science, Peking University, No.5 Yiheyuan Road Haidian District, Beijing 100871, PR China.
| | - Xiaolong Zou
- School of Electronics Engineering and Computer Science, Peking University, No.5 Yiheyuan Road Haidian District, Beijing 100871, PR China; School of Psychological and Cognitive Sciences, IDG/McGovern Institute for Brain Research, PKU-Tsinghua Center for Life Sciences, Peking University, No.5 Yiheyuan Road Haidian District, Beijing 100871, PR China.
| | - Zilong Ji
- School of Electronics Engineering and Computer Science, Peking University, No.5 Yiheyuan Road Haidian District, Beijing 100871, PR China; School of Psychological and Cognitive Sciences, IDG/McGovern Institute for Brain Research, PKU-Tsinghua Center for Life Sciences, Peking University, No.5 Yiheyuan Road Haidian District, Beijing 100871, PR China.
| | - Tiejun Huang
- School of Electronics Engineering and Computer Science, Peking University, No.5 Yiheyuan Road Haidian District, Beijing 100871, PR China.
| | - Si Wu
- School of Electronics Engineering and Computer Science, Peking University, No.5 Yiheyuan Road Haidian District, Beijing 100871, PR China; School of Psychological and Cognitive Sciences, IDG/McGovern Institute for Brain Research, PKU-Tsinghua Center for Life Sciences, Peking University, No.5 Yiheyuan Road Haidian District, Beijing 100871, PR China.
| | - Yuanyuan Mi
- Center for Neurointelligence, School of Medicine, Chongqing University, No.174 Shazhengjie, Shapingba, Chongqing 400044, PR China; AI Research Center, Peng Cheng Laboratory, No.2, Xingke First Street, Nanshan District, Shenzhen 518005, PR China.
| |
Collapse
|
8
|
Yang S, Gao T, Wang J, Deng B, Lansdell B, Linares-Barranco B. Efficient Spike-Driven Learning With Dendritic Event-Based Processing. Front Neurosci 2021; 15:601109. [PMID: 33679295 PMCID: PMC7933681 DOI: 10.3389/fnins.2021.601109] [Citation(s) in RCA: 60] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2020] [Accepted: 01/21/2021] [Indexed: 11/22/2022] Open
Abstract
A critical challenge in neuromorphic computing is to present computationally efficient algorithms of learning. When implementing gradient-based learning, error information must be routed through the network, such that each neuron knows its contribution to output, and thus how to adjust its weight. This is known as the credit assignment problem. Exactly implementing a solution like backpropagation involves weight sharing, which requires additional bandwidth and computations in a neuromorphic system. Instead, models of learning from neuroscience can provide inspiration for how to communicate error information efficiently, without weight sharing. Here we present a novel dendritic event-based processing (DEP) algorithm, using a two-compartment leaky integrate-and-fire neuron with partially segregated dendrites that effectively solves the credit assignment problem. In order to optimize the proposed algorithm, a dynamic fixed-point representation method and piecewise linear approximation approach are presented, while the synaptic events are binarized during learning. The presented optimization makes the proposed DEP algorithm very suitable for implementation in digital or mixed-signal neuromorphic hardware. The experimental results show that spiking representations can rapidly learn, achieving high performance by using the proposed DEP algorithm. We find the learning capability is affected by the degree of dendritic segregation, and the form of synaptic feedback connections. This study provides a bridge between the biological learning and neuromorphic learning, and is meaningful for the real-time applications in the field of artificial intelligence.
Collapse
Affiliation(s)
- Shuangming Yang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Tian Gao
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Jiang Wang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Bin Deng
- School of Electrical and Information Engineering, Tianjin University, Tianjin, China
| | - Benjamin Lansdell
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, United States
| | | |
Collapse
|
9
|
Martinez De La Cruz B, Markus R, Malla S, Haig MI, Gell C, Sang F, Bellows E, Sherif MA, McLean D, Lourdusamy A, Self T, Bodi Z, Smith S, Fay M, Macdonald IA, Fray R, Knight HM. Modifying the m 6A brain methylome by ALKBH5-mediated demethylation: a new contender for synaptic tagging. Mol Psychiatry 2021; 26:7141-7153. [PMID: 34663904 PMCID: PMC8872986 DOI: 10.1038/s41380-021-01282-z] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/27/2020] [Revised: 08/02/2021] [Accepted: 08/25/2021] [Indexed: 02/08/2023]
Abstract
Synaptic plasticity processes, which underlie learning and memory formation, require RNA to be translated local to synapses. The synaptic tagging hypothesis has previously been proposed to explain how mRNAs are available at specific activated synapses. However how RNA is regulated, and which transcripts are silenced or processed as part of the tagging process is still unknown. Modification of RNA by N6-methyladenosine (m6A/m) influences the cellular fate of mRNA. Here, by advanced microscopy, we showed that m6A demethylation by the eraser protein ALKBH5 occurs at active synaptic ribosomes and at synapses during short term plasticity. We demonstrated that at activated glutamatergic post-synaptic sites, both the YTHDF1 and YTHDF3 reader and the ALKBH5 eraser proteins increase in co-localisation to m6A-modified RNAs; but only the readers showed high co-localisation to modified RNAs during late-stage plasticity. The YTHDF1 and YTHFDF3 readers also exhibited differential roles during synaptic maturation suggesting that temporal and subcellular abundance may determine specific function. m6A-sequencing of human parahippocampus brain tissue revealed distinct white and grey matter m6A methylome profiles indicating that cellular context is a fundamental factor dictating regulated pathways. However, in both neuronal and glial cell-rich tissue, m6A effector proteins are themselves modified and m6A epitranscriptional and posttranslational modification processes coregulate protein cascades. We hypothesise that the availability m6A effector protein machinery in conjunction with RNA modification, may be important in the formation of condensed synaptic nanodomain assemblies through liquid-liquid phase separation. Our findings support that m6A demethylation by ALKBH5 is an intrinsic component of the synaptic tagging hypothesis and a molecular switch which leads to alterations in the RNA methylome, synaptic dysfunction and potentially reversible disease states.
Collapse
Affiliation(s)
- Braulio Martinez De La Cruz
- grid.4563.40000 0004 1936 8868Division of Cells, Organisms and Molecular Genetics, School of Life Sciences, University of Nottingham, Nottingham, UK ,grid.415971.f0000 0004 0605 8588Present Address: MRC Laboratory of Molecular Cell Biology, UCL, London, UK
| | - Robert Markus
- grid.4563.40000 0004 1936 8868School of Life Sciences Imaging Facility, University of Nottingham, Nottingham, UK
| | - Sunir Malla
- grid.4563.40000 0004 1936 8868Deep Seq: Next Generation Sequencing Facility, University of Nottingham, Nottingham, UK
| | - Maria Isabel Haig
- grid.4563.40000 0004 1936 8868Division of Cells, Organisms and Molecular Genetics, School of Life Sciences, University of Nottingham, Nottingham, UK
| | - Chris Gell
- grid.4563.40000 0004 1936 8868School of Life Sciences Imaging Facility, University of Nottingham, Nottingham, UK
| | - Fei Sang
- grid.4563.40000 0004 1936 8868Deep Seq: Next Generation Sequencing Facility, University of Nottingham, Nottingham, UK
| | - Eleanor Bellows
- grid.4563.40000 0004 1936 8868Division of Cells, Organisms and Molecular Genetics, School of Life Sciences, University of Nottingham, Nottingham, UK
| | - Mahmoud Awad Sherif
- grid.4563.40000 0004 1936 8868Division of Cells, Organisms and Molecular Genetics, School of Life Sciences, University of Nottingham, Nottingham, UK
| | - Denise McLean
- grid.4563.40000 0004 1936 8868School of Life Sciences Imaging Facility, University of Nottingham, Nottingham, UK
| | - Anbarasu Lourdusamy
- grid.4563.40000 0004 1936 8868Children’s Brain Tumour Research Centre, School of Medicine, University of Nottingham, Nottingham, UK
| | - Tim Self
- grid.4563.40000 0004 1936 8868School of Life Sciences Imaging Facility, University of Nottingham, Nottingham, UK
| | - Zsuzsanna Bodi
- grid.4563.40000 0004 1936 8868Division of Plant Sciences, School of Biosciences, University of Nottingham, Nottingham, UK
| | - Stuart Smith
- grid.4563.40000 0004 1936 8868Children’s Brain Tumour Research Centre, School of Medicine, University of Nottingham, Nottingham, UK
| | - Michael Fay
- grid.4563.40000 0004 1936 8868Nanoscale and Microscale Research Centre, University of Nottingham, Nottingham, UK
| | - Ian A. Macdonald
- grid.4563.40000 0004 1936 8868Division of Physiology, Pharmacology and Neuroscience, School of Life Sciences, University of Nottingham, Nottingham, UK
| | - Rupert Fray
- grid.4563.40000 0004 1936 8868Division of Plant Sciences, School of Biosciences, University of Nottingham, Nottingham, UK
| | - Helen Miranda Knight
- Division of Cells, Organisms and Molecular Genetics, School of Life Sciences, University of Nottingham, Nottingham, UK.
| |
Collapse
|
10
|
Kruijne W, Bohte SM, Roelfsema PR, Olivers CNL. Flexible Working Memory Through Selective Gating and Attentional Tagging. Neural Comput 2020; 33:1-40. [PMID: 33080159 DOI: 10.1162/neco_a_01339] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/02/2023]
Abstract
Working memory is essential: it serves to guide intelligent behavior of humans and nonhuman primates when task-relevant stimuli are no longer present to the senses. Moreover, complex tasks often require that multiple working memory representations can be flexibly and independently maintained, prioritized, and updated according to changing task demands. Thus far, neural network models of working memory have been unable to offer an integrative account of how such control mechanisms can be acquired in a biologically plausible manner. Here, we present WorkMATe, a neural network architecture that models cognitive control over working memory content and learns the appropriate control operations needed to solve complex working memory tasks. Key components of the model include a gated memory circuit that is controlled by internal actions, encoding sensory information through untrained connections, and a neural circuit that matches sensory inputs to memory content. The network is trained by means of a biologically plausible reinforcement learning rule that relies on attentional feedback and reward prediction errors to guide synaptic updates. We demonstrate that the model successfully acquires policies to solve classical working memory tasks, such as delayed recognition and delayed pro-saccade/anti-saccade tasks. In addition, the model solves much more complex tasks, including the hierarchical 12-AX task or the ABAB ordered recognition task, both of which demand an agent to independently store and updated multiple items separately in memory. Furthermore, the control strategies that the model acquires for these tasks subsequently generalize to new task contexts with novel stimuli, thus bringing symbolic production rule qualities to a neural network architecture. As such, WorkMATe provides a new solution for the neural implementation of flexible memory control.
Collapse
Affiliation(s)
- Wouter Kruijne
- Faculty of Behavior and Movement Sciences, Vrije Universiteit Amsterdam, 1081 BT Amsterdam, Noord Holland, The Netherlands
| | - Sander M Bohte
- Machine Learning Group, Centrum voor Wiskunde & Informatica, 1098 XG Amsterdam, Noord Holland, The Netherlands; Swammerdam Institute of Life Sciences, University of Amsterdam, 1098 XH Amsterdam, Noord Holland, The Netherlands; and Department of Computer Science, Rijksuniversiteit Groningen, 9747 AG Groningen, The Netherlands
| | - Pieter R Roelfsema
- Department of Vision & Cognition, Netherlands Institute for Neuroscience, 1105BA Amsterdam, Noord Holland, The Netherlands; Department of Integrative Neurophysiology, Center for Neurogenomics and Cognitive Research, Vrije Universiteit Amsterdam, 1981 HV Amsterdam, Noord Holland, The Netherlands; and Department of Computer Science, Rijksuniversiteit Groningen, 9747 AG Groningen, The Netherlands
| | - Christian N L Olivers
- Faculty of Behavior and Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, Noord Holland, The Netherlands, Department of Psychiatry, Academic Medical Center, Amsterdam, The Netherlands
| |
Collapse
|
11
|
Olivers CN, Roelfsema PR. Attention for action in visual working memory. Cortex 2020; 131:179-194. [DOI: 10.1016/j.cortex.2020.07.011] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2020] [Revised: 06/22/2020] [Accepted: 07/14/2020] [Indexed: 12/27/2022]
|
12
|
Pals M, Stewart TC, Akyürek EG, Borst JP. A functional spiking-neuron model of activity-silent working memory in humans based on calcium-mediated short-term synaptic plasticity. PLoS Comput Biol 2020; 16:e1007936. [PMID: 32516337 PMCID: PMC7282629 DOI: 10.1371/journal.pcbi.1007936] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2019] [Accepted: 05/07/2020] [Indexed: 11/19/2022] Open
Abstract
In this paper, we present a functional spiking-neuron model of human working memory (WM). This model combines neural firing for encoding of information with activity-silent maintenance. While it used to be widely assumed that information in WM is maintained through persistent recurrent activity, recent studies have shown that information can be maintained without persistent firing; instead, information can be stored in activity-silent states. A candidate mechanism underlying this type of storage is short-term synaptic plasticity (STSP), by which the strength of connections between neurons rapidly changes to encode new information. To demonstrate that STSP can lead to functional behavior, we integrated STSP by means of calcium-mediated synaptic facilitation in a large-scale spiking-neuron model and added a decision mechanism. The model was used to simulate a recent study that measured behavior and EEG activity of participants in three delayed-response tasks. In these tasks, one or two visual gratings had to be maintained in WM, and compared to subsequent probes. The original study demonstrated that WM contents and its priority status could be decoded from neural activity elicited by a task-irrelevant stimulus displayed during the activity-silent maintenance period. In support of our model, we show that it can perform these tasks, and that both its behavior as well as its neural representations are in agreement with the human data. We conclude that information in WM can be effectively maintained in activity-silent states by means of calcium-mediated STSP. Mentally maintaining information for short periods of time in working memory is crucial for human adaptive behavior. It was recently shown that the human brain does not only store information through neural firing–as was widely believed–but also maintains information in activity-silent states. Here, we present a detailed neural model of how this could happen in our brain through short-term synaptic plasticity: rapidly adapting the connection strengths between neurons in response to incoming information. By reactivating the adapted network, the stored information can be read out later. We show that our model can perform three working memory tasks as accurately as human participants can, while using similar mental representations. We conclude that our model is a plausible and effective neural implementation of human working memory.
Collapse
Affiliation(s)
- Matthijs Pals
- Bernoulli Institute, University of Groningen, Groningen, The Netherlands
| | - Terrence C. Stewart
- Centre for Theoretical Neuroscience, University of Waterloo, Waterloo, Ontario, Canada
| | - Elkan G. Akyürek
- Department of Experimental Psychology, University of Groningen, Groningen, The Netherlands
| | - Jelmer P. Borst
- Bernoulli Institute, University of Groningen, Groningen, The Netherlands
- Groningen Cognitive Systems and Materials Center, University of Groningen, Groningen, The Netherlands
- * E-mail:
| |
Collapse
|
13
|
Azimi M, Oemisch M, Womelsdorf T. Dissociation of nicotinic α7 and α4/β2 sub-receptor agonists for enhancing learning and attentional filtering in nonhuman primates. Psychopharmacology (Berl) 2020; 237:997-1010. [PMID: 31865424 DOI: 10.1007/s00213-019-05430-w] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/07/2018] [Accepted: 12/11/2019] [Indexed: 12/22/2022]
Abstract
RATIONALE Nicotinic acetylcholine receptors (nAChRs) modulate attention, memory, and higher executive functioning, but it is unclear how nACh sub-receptors mediate different mechanisms supporting these functions. OBJECTIVES We investigated whether selective agonists for the alpha-7 nAChR versus the alpha-4/beta-2 nAChR have unique functional contributions for value learning and attentional filtering of distractors in the nonhuman primate. METHODS Two adult rhesus macaque monkeys performed reversal learning following systemic administration of either the alpha-7 nAChR agonist PHA-543613 or the alpha-4/beta-2 nAChR agonist ABT-089 or a vehicle control. Behavioral analysis quantified performance accuracy, speed of processing, reversal learning speed, the control of distractor interference, perseveration tendencies, and motivation. RESULTS We found that the alpha-7 nAChR agonist PHA-543613 enhanced the learning speed of feature values but did not modulate how salient distracting information was filtered from ongoing choice processes. In contrast, the selective alpha-4/beta-2 nAChR agonist ABT-089 did not affect learning speed but reduced distractibility. This dissociation was dose-dependent and evident in the absence of systematic changes in overall performance, reward intake, motivation to perform the task, perseveration tendencies, or reaction times. CONCLUSIONS These results suggest nicotinic sub-receptor specific mechanisms consistent with (1) alpha-4/beta-2 nAChR specific amplification of cholinergic transients in prefrontal cortex linked to enhanced cue detection in light of interferences, and (2) alpha-7 nAChR specific activation prolonging cholinergic transients, which could facilitate subjects to follow-through with newly established attentional strategies when outcome contingencies change. These insights will be critical for developing function-specific drugs alleviating attention and learning deficits in neuro-psychiatric diseases.
Collapse
Affiliation(s)
- Marzyeh Azimi
- Department of Biology, Centre for Vision Research, York University, Toronto, Ontario, M6J 1P3, Canada
| | - Mariann Oemisch
- Department of Biology, Centre for Vision Research, York University, Toronto, Ontario, M6J 1P3, Canada.,The Zanvyl Krieger Mind/Brain Institute, Department of Neuroscience, Johns Hopkins University, Baltimore, MD, 21218, USA
| | - Thilo Womelsdorf
- Department of Biology, Centre for Vision Research, York University, Toronto, Ontario, M6J 1P3, Canada. .,Department of Psychology, Vanderbilt University, PMB 407817, 2301, Vanderbilt Place, Nashville, TN, 37240-7817, USA.
| |
Collapse
|
14
|
Mashour GA, Roelfsema P, Changeux JP, Dehaene S. Conscious Processing and the Global Neuronal Workspace Hypothesis. Neuron 2020; 105:776-798. [PMID: 32135090 PMCID: PMC8770991 DOI: 10.1016/j.neuron.2020.01.026] [Citation(s) in RCA: 364] [Impact Index Per Article: 91.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2019] [Revised: 10/31/2019] [Accepted: 01/22/2020] [Indexed: 10/24/2022]
Abstract
We review the central tenets and neuroanatomical basis of the global neuronal workspace (GNW) hypothesis, which attempts to account for the main scientific observations regarding the elementary mechanisms of conscious processing in the human brain. The GNW hypothesis proposes that, in the conscious state, a non-linear network ignition associated with recurrent processing amplifies and sustains a neural representation, allowing the corresponding information to be globally accessed by local processors. We examine this hypothesis in light of recent data that contrast brain activity evoked by either conscious or non-conscious contents, as well as during conscious or non-conscious states, particularly general anesthesia. We also discuss the relationship between the intertwined concepts of conscious processing, attention, and working memory.
Collapse
Affiliation(s)
- George A Mashour
- Center for Consciousness Science, Neuroscience Graduate Program, and Department of Anesthesiology, University of Michigan, Ann Arbor, MI, USA
| | - Pieter Roelfsema
- Department of Vision & Cognition, Netherlands Institute for Neuroscience, Meibergdreef 47, 1105 BA, Amsterdam, the Netherlands; Department of Integrative Neurophysiology, Center for Neurogenomics and Cognitive Research, VU University, Amsterdam, the Netherlands; Department of Psychiatry, Academic Medical Center, Amsterdam, the Netherlands
| | - Jean-Pierre Changeux
- CNRS UMR 3571, Institut Pasteur, 75724 Paris, France; Collège de France, 11 Place Marcelin Berthelot, 75005 Paris, France; Kavli Institute for Brain & Mind, University of California, San Diego, La Jolla, CA, USA.
| | - Stanislas Dehaene
- Collège de France, 11 Place Marcelin Berthelot, 75005 Paris, France; Cognitive Neuroimaging Unit, CEA, INSERM, Université Paris-Sud, Université Paris-Saclay, NeuroSpin Center, 91191 Gif/Yvette, France.
| |
Collapse
|
15
|
Roelfsema PR, Holtmaat A. Control of synaptic plasticity in deep cortical networks. Nat Rev Neurosci 2019; 19:166-180. [PMID: 29449713 DOI: 10.1038/nrn.2018.6] [Citation(s) in RCA: 105] [Impact Index Per Article: 21.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/07/2023]
Abstract
Humans and many other animals have an enormous capacity to learn about sensory stimuli and to master new skills. However, many of the mechanisms that enable us to learn remain to be understood. One of the greatest challenges of systems neuroscience is to explain how synaptic connections change to support maximally adaptive behaviour. Here, we provide an overview of factors that determine the change in the strength of synapses, with a focus on synaptic plasticity in sensory cortices. We review the influence of neuromodulators and feedback connections in synaptic plasticity and suggest a specific framework in which these factors can interact to improve the functioning of the entire network.
Collapse
Affiliation(s)
- Pieter R Roelfsema
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, Royal Netherlands Academy of Arts and Sciences, Amsterdam, Netherlands.,Department of Integrative Neurophysiology, Center for Neurogenomics and Cognitive Research, VU University, Amsterdam, Netherlands.,Psychiatry Department, Academic Medical Center, Amsterdam, Netherlands
| | - Anthony Holtmaat
- Department of Basic Neurosciences, Geneva Neuroscience Center, Faculty of Medicine, University of Geneva, Geneva, Switzerland
| |
Collapse
|
16
|
|
17
|
Jordan J, Weidel P, Morrison A. A Closed-Loop Toolchain for Neural Network Simulations of Learning Autonomous Agents. Front Comput Neurosci 2019; 13:46. [PMID: 31427939 PMCID: PMC6687756 DOI: 10.3389/fncom.2019.00046] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2019] [Accepted: 06/25/2019] [Indexed: 11/17/2022] Open
Abstract
Neural network simulation is an important tool for generating and evaluating hypotheses on the structure, dynamics, and function of neural circuits. For scientific questions addressing organisms operating autonomously in their environments, in particular where learning is involved, it is crucial to be able to operate such simulations in a closed-loop fashion. In such a set-up, the neural agent continuously receives sensory stimuli from the environment and provides motor signals that manipulate the environment or move the agent within it. So far, most studies requiring such functionality have been conducted with custom simulation scripts and manually implemented tasks. This makes it difficult for other researchers to reproduce and build upon previous work and nearly impossible to compare the performance of different learning architectures. In this work, we present a novel approach to solve this problem, connecting benchmark tools from the field of machine learning and state-of-the-art neural network simulators from computational neuroscience. The resulting toolchain enables researchers in both fields to make use of well-tested high-performance simulation software supporting biologically plausible neuron, synapse and network models and allows them to evaluate and compare their approach on the basis of standardized environments with various levels of complexity. We demonstrate the functionality of the toolchain by implementing a neuronal actor-critic architecture for reinforcement learning in the NEST simulator and successfully training it on two different environments from the OpenAI Gym. We compare its performance to a previously suggested neural network model of reinforcement learning in the basal ganglia and a generic Q-learning algorithm.
Collapse
Affiliation(s)
- Jakob Jordan
- Department of Physiology, University of Bern, Bern, Switzerland
- Institute of Neuroscience and Medicine (INM-6) & Institute for Advanced Simulation (IAS-6) & JARA-Institute Brain Structure Function Relationship (JBI 1/INM-10), Research Centre Jülich, Jülich, Germany
| | - Philipp Weidel
- Institute of Neuroscience and Medicine (INM-6) & Institute for Advanced Simulation (IAS-6) & JARA-Institute Brain Structure Function Relationship (JBI 1/INM-10), Research Centre Jülich, Jülich, Germany
- aiCTX, Zurich, Switzerland
- Department of Computer Science, RWTH Aachen University, Aachen, Germany
| | - Abigail Morrison
- Institute of Neuroscience and Medicine (INM-6) & Institute for Advanced Simulation (IAS-6) & JARA-Institute Brain Structure Function Relationship (JBI 1/INM-10), Research Centre Jülich, Jülich, Germany
- Faculty of Psychology, Institute of Cognitive Neuroscience, Ruhr-University Bochum, Bochum, Germany
| |
Collapse
|
18
|
Freedman DJ, Ibos G. An Integrative Framework for Sensory, Motor, and Cognitive Functions of the Posterior Parietal Cortex. Neuron 2019; 97:1219-1234. [PMID: 29566792 DOI: 10.1016/j.neuron.2018.01.044] [Citation(s) in RCA: 69] [Impact Index Per Article: 13.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2017] [Revised: 01/12/2018] [Accepted: 01/23/2018] [Indexed: 11/28/2022]
Abstract
Throughout the history of modern neuroscience, the parietal cortex has been associated with a wide array of sensory, motor, and cognitive functions. The use of non-human primates as a model organism has been instrumental in our current understanding of how areas in the posterior parietal cortex (PPC) modulate our perception and influence our behavior. In this Perspective, we highlight a series of influential studies over the last five decades examining the role of the PPC in visual perception and motor planning. We also integrate long-standing views of PPC functions with more recent evidence to propose a more general model framework to explain integrative sensory, motor, and cognitive functions of the PPC.
Collapse
Affiliation(s)
- David J Freedman
- Department of Neurobiology, The University of Chicago, Chicago, IL 60637, USA; Grossman Institute for Neuroscience, Quantitative Biology and Human Behavior, The University of Chicago, Chicago, IL 60637, USA.
| | - Guilhem Ibos
- Department of Neurobiology, The University of Chicago, Chicago, IL 60637, USA; Institut de Neuroscience de la Timone, UMR 7289 CNRS & Aix-Marseille Université, Marseille, France.
| |
Collapse
|
19
|
Illing B, Gerstner W, Brea J. Biologically plausible deep learning - But how far can we go with shallow networks? Neural Netw 2019; 118:90-101. [PMID: 31254771 DOI: 10.1016/j.neunet.2019.06.001] [Citation(s) in RCA: 30] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2019] [Revised: 05/29/2019] [Accepted: 06/02/2019] [Indexed: 11/17/2022]
Abstract
Training deep neural networks with the error backpropagation algorithm is considered implausible from a biological perspective. Numerous recent publications suggest elaborate models for biologically plausible variants of deep learning, typically defining success as reaching around 98% test accuracy on the MNIST data set. Here, we investigate how far we can go on digit (MNIST) and object (CIFAR10) classification with biologically plausible, local learning rules in a network with one hidden layer and a single readout layer. The hidden layer weights are either fixed (random or random Gabor filters) or trained with unsupervised methods (Principal/Independent Component Analysis or Sparse Coding) that can be implemented by local learning rules. The readout layer is trained with a supervised, local learning rule. We first implement these models with rate neurons. This comparison reveals, first, that unsupervised learning does not lead to better performance than fixed random projections or Gabor filters for large hidden layers. Second, networks with localized receptive fields perform significantly better than networks with all-to-all connectivity and can reach backpropagation performance on MNIST. We then implement two of the networks - fixed, localized, random & random Gabor filters in the hidden layer - with spiking leaky integrate-and-fire neurons and spike timing dependent plasticity to train the readout layer. These spiking models achieve >98.2% test accuracy on MNIST, which is close to the performance of rate networks with one hidden layer trained with backpropagation. The performance of our shallow network models is comparable to most current biologically plausible models of deep learning. Furthermore, our results with a shallow spiking network provide an important reference and suggest the use of data sets other than MNIST for testing the performance of future models of biologically plausible deep learning.
Collapse
Affiliation(s)
- Bernd Illing
- School of Computer and Communication Science & School of Life Science, EPFL, 1015 Lausanne, Switzerland.
| | - Wulfram Gerstner
- School of Computer and Communication Science & School of Life Science, EPFL, 1015 Lausanne, Switzerland
| | - Johanni Brea
- School of Computer and Communication Science & School of Life Science, EPFL, 1015 Lausanne, Switzerland
| |
Collapse
|
20
|
Oemisch M, Westendorff S, Azimi M, Hassani SA, Ardid S, Tiesinga P, Womelsdorf T. Feature-specific prediction errors and surprise across macaque fronto-striatal circuits. Nat Commun 2019; 10:176. [PMID: 30635579 PMCID: PMC6329800 DOI: 10.1038/s41467-018-08184-9] [Citation(s) in RCA: 33] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2018] [Accepted: 12/20/2018] [Indexed: 01/23/2023] Open
Abstract
To adjust expectations efficiently, prediction errors need to be associated with the precise features that gave rise to the unexpected outcome, but this credit assignment may be problematic if stimuli differ on multiple dimensions and it is ambiguous which feature dimension caused the outcome. Here, we report a potential solution: neurons in four recorded areas of the anterior fronto-striatal networks encode prediction errors that are specific to feature values of different dimensions of attended multidimensional stimuli. The most ubiquitous prediction error occurred for the reward-relevant dimension. Feature-specific prediction error signals a) emerge on average shortly after non-specific prediction error signals, b) arise earliest in the anterior cingulate cortex and later in dorsolateral prefrontal cortex, caudate and ventral striatum, and c) contribute to feature-based stimulus selection after learning. Thus, a widely-distributed feature-specific eligibility trace may be used to update synaptic weights for improved feature-based attention. In order to adjust expectations efficiently, prediction errors need to be associated with the features that gave rise to the unexpected outcome. Here, the authors show that neurons in anterior fronto-striatal networks encode prediction errors that are specific to feature values of different stimulus dimensions.
Collapse
Affiliation(s)
- Mariann Oemisch
- Department of Biology, Centre for Vision Research, York University, 4700 Keele Street, Toronto, ON, M6J 1P3, Canada. .,Department of Neuroscience, Yale University School of Medicine, New Haven, CT, 06510, USA.
| | - Stephanie Westendorff
- Department of Biology, Centre for Vision Research, York University, 4700 Keele Street, Toronto, ON, M6J 1P3, Canada.,Institute of Neurobiology, University of Tübingen, Tübingen, 72076, Germany
| | - Marzyeh Azimi
- Department of Biology, Centre for Vision Research, York University, 4700 Keele Street, Toronto, ON, M6J 1P3, Canada
| | - Seyed Alireza Hassani
- Department of Biology, Centre for Vision Research, York University, 4700 Keele Street, Toronto, ON, M6J 1P3, Canada.,Department of Psychology, Vanderbilt University, Nashville, TN, 37240, USA
| | - Salva Ardid
- Department of Mathematics and Statistics, Boston University, Boston, MA, 02215, USA
| | - Paul Tiesinga
- Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, 6525 EN, Netherlands
| | - Thilo Womelsdorf
- Department of Biology, Centre for Vision Research, York University, 4700 Keele Street, Toronto, ON, M6J 1P3, Canada. .,Department of Psychology, Vanderbilt University, Nashville, TN, 37240, USA.
| |
Collapse
|
21
|
Richards BA, Lillicrap TP. Dendritic solutions to the credit assignment problem. Curr Opin Neurobiol 2018; 54:28-36. [PMID: 30205266 DOI: 10.1016/j.conb.2018.08.003] [Citation(s) in RCA: 50] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2018] [Revised: 07/19/2018] [Accepted: 08/07/2018] [Indexed: 11/27/2022]
Abstract
Guaranteeing that synaptic plasticity leads to effective learning requires a means for assigning credit to each neuron for its contribution to behavior. The 'credit assignment problem' refers to the fact that credit assignment is non-trivial in hierarchical networks with multiple stages of processing. One difficulty is that if credit signals are integrated with other inputs, then it is hard for synaptic plasticity rules to distinguish credit-related activity from non-credit-related activity. A potential solution is to use the spatial layout and non-linear properties of dendrites to distinguish credit signals from other inputs. In cortical pyramidal neurons, evidence hints that top-down feedback signals are integrated in the distal apical dendrites and have a distinct impact on spike-firing and synaptic plasticity. This suggests that the distal apical dendrites of pyramidal neurons help the brain to solve the credit assignment problem.
Collapse
Affiliation(s)
- Blake A Richards
- Department of Biological Sciences, University of Toronto Scarborough, Toronto, ON, Canada; Department of Cell and Systems Biology, University of Toronto, Toronto, ON, Canada; Learning in Machines and Brains Program, Canadian Institute for Advanced Research, Toronto, ON, Canada
| | | |
Collapse
|
22
|
Martinolli M, Gerstner W, Gilra A. Multi-Timescale Memory Dynamics Extend Task Repertoire in a Reinforcement Learning Network With Attention-Gated Memory. Front Comput Neurosci 2018; 12:50. [PMID: 30061819 PMCID: PMC6055065 DOI: 10.3389/fncom.2018.00050] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2018] [Accepted: 06/18/2018] [Indexed: 11/13/2022] Open
Abstract
The interplay of reinforcement learning and memory is at the core of several recent neural network models, such as the Attention-Gated MEmory Tagging (AuGMEnT) model. While successful at various animal learning tasks, we find that the AuGMEnT network is unable to cope with some hierarchical tasks, where higher-level stimuli have to be maintained over a long time, while lower-level stimuli need to be remembered and forgotten over a shorter timescale. To overcome this limitation, we introduce a hybrid AuGMEnT, with leaky (or short-timescale) and non-leaky (or long-timescale) memory units, that allows the exchange of low-level information while maintaining high-level one. We test the performance of the hybrid AuGMEnT network on two cognitive reference tasks, sequence prediction and 12AX.
Collapse
Affiliation(s)
- Marco Martinolli
- School of Computer and Communication Sciences, School of Life Sciences, Brain-Mind Institute, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Wulfram Gerstner
- School of Computer and Communication Sciences, School of Life Sciences, Brain-Mind Institute, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Aditya Gilra
- School of Computer and Communication Sciences, School of Life Sciences, Brain-Mind Institute, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| |
Collapse
|
23
|
Gerstner W, Lehmann M, Liakoni V, Corneil D, Brea J. Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules. Front Neural Circuits 2018; 12:53. [PMID: 30108488 PMCID: PMC6079224 DOI: 10.3389/fncir.2018.00053] [Citation(s) in RCA: 112] [Impact Index Per Article: 18.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2018] [Accepted: 06/19/2018] [Indexed: 11/13/2022] Open
Abstract
Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules.
Collapse
Affiliation(s)
- Wulfram Gerstner
- School of Computer Science and School of Life Sciences, École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | | | | | | | | |
Collapse
|
24
|
Khan AG, Hofer SB. Contextual signals in visual cortex. Curr Opin Neurobiol 2018; 52:131-138. [PMID: 29883940 DOI: 10.1016/j.conb.2018.05.003] [Citation(s) in RCA: 46] [Impact Index Per Article: 7.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2018] [Accepted: 05/11/2018] [Indexed: 11/15/2022]
Abstract
Vision is an active process. What we perceive strongly depends on our actions, intentions and expectations. During visual processing, these internal signals therefore need to be integrated with the visual information from the retina. The mechanisms of how this is achieved by the visual system are still poorly understood. Advances in recording and manipulating neuronal activity in specific cell types and axonal projections together with tools for circuit tracing are beginning to shed light on the neuronal circuit mechanisms of how internal, contextual signals shape sensory representations. Here we review recent work, primarily in mice, that has advanced our understanding of these processes, focusing on contextual signals related to locomotion, behavioural relevance and predictions.
Collapse
Affiliation(s)
- Adil G Khan
- Centre for Developmental Neurobiology, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Sonja B Hofer
- Sainsbury Wellcome Centre for Neural Circuits and Behaviour, University College London, London, UK.
| |
Collapse
|
25
|
The influence of attention and reward on the learning of stimulus-response associations. Sci Rep 2017; 7:9036. [PMID: 28831043 PMCID: PMC5567207 DOI: 10.1038/s41598-017-08200-w] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2016] [Accepted: 07/06/2017] [Indexed: 12/24/2022] Open
Abstract
We can learn new tasks by listening to a teacher, but we can also learn by trial-and-error. Here, we investigate the factors that determine how participants learn new stimulus-response mappings by trial-and-error. Does learning in human observers comply with reinforcement learning theories, which describe how subjects learn from rewards and punishments? If yes, what is the influence of selective attention in the learning process? We developed a novel redundant-relevant learning paradigm to examine the conjoint influence of attention and reward feedback. We found that subjects only learned stimulus-response mappings for attended shapes, even when unattended shapes were equally informative. Reward magnitude also influenced learning, an effect that was stronger for attended than for non-attended shapes and that carried over to a subsequent visual search task. Our results provide insights into how attention and reward jointly determine how we learn. They support the powerful learning rules that capitalize on the conjoint influence of these two factors on neuronal plasticity.
Collapse
|
26
|
Christophel TB, Klink PC, Spitzer B, Roelfsema PR, Haynes JD. The Distributed Nature of Working Memory. Trends Cogn Sci 2017; 21:111-124. [PMID: 28063661 DOI: 10.1016/j.tics.2016.12.007] [Citation(s) in RCA: 412] [Impact Index Per Article: 58.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2016] [Revised: 12/03/2016] [Accepted: 12/07/2016] [Indexed: 12/25/2022]
Abstract
Studies in humans and non-human primates have provided evidence for storage of working memory contents in multiple regions ranging from sensory to parietal and prefrontal cortex. We discuss potential explanations for these distributed representations: (i) features in sensory regions versus prefrontal cortex differ in the level of abstractness and generalizability; and (ii) features in prefrontal cortex reflect representations that are transformed for guidance of upcoming behavioral actions. We propose that the propensity to produce persistent activity is a general feature of cortical networks. Future studies may have to shift focus from asking where working memory can be observed in the brain to how a range of specialized brain areas together transform sensory information into a delayed behavioral response.
Collapse
Affiliation(s)
- Thomas B Christophel
- Bernstein Center for Computational Neuroscience, Charité Universitätsmedizin, Berlin, Germany; Berlin Center for Advanced Neuroimaging, Charité Universitätsmedizin, Berlin, Germany; Clinic for Neurology, Charité Universitätsmedizin, Berlin, Germany.
| | - P Christiaan Klink
- Department of Neuromodulation & Behaviour, Netherlands Institute for Neuroscience, Amsterdam, the Netherlands; Department of Vision & Cognition, Netherlands Institute for Neuroscience, Amsterdam, the Netherlands; Department of Psychiatry, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands
| | - Bernhard Spitzer
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Pieter R Roelfsema
- Department of Vision & Cognition, Netherlands Institute for Neuroscience, Amsterdam, the Netherlands; Department of Psychiatry, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands; Department of Integrative Neurophysiology, Centre for Neurogenomics and Cognitive Research, VU University, Amsterdam, The Netherlands
| | - John-Dylan Haynes
- Bernstein Center for Computational Neuroscience, Charité Universitätsmedizin, Berlin, Germany; Berlin Center for Advanced Neuroimaging, Charité Universitätsmedizin, Berlin, Germany; Clinic for Neurology, Charité Universitätsmedizin, Berlin, Germany; Berlin School of Mind and Brain, Humboldt Universität, Berlin, Germany; Cluster of Excellence NeuroCure, Charité Universitätsmedizin, Berlin, Germany; Department of Psychology, Humboldt Universität zu Berlin, Berlin, Germany
| |
Collapse
|
27
|
Petro LS, Muckli L. The laminar integration of sensory inputs with feedback signals in human cortex. Brain Cogn 2016; 112:54-57. [PMID: 27814926 PMCID: PMC5312781 DOI: 10.1016/j.bandc.2016.06.007] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2015] [Revised: 06/23/2016] [Accepted: 06/24/2016] [Indexed: 11/25/2022]
Abstract
Understanding how the cortex integrates feedback and feedforward signals is central to understanding brain function. The data-driven framework of apical amplification which is hypothesized to have a central role in cognition is highlighted. Human neuroimaging data provides evidence for layer-specific cortical feedback relevant for theories of predictive feedback.
The cortex constitutes the largest area of the human brain. Yet we have only a basic understanding of how the cortex performs one vital function: the integration of sensory signals (carried by feedforward pathways) with internal representations (carried by feedback pathways). A multi-scale, multi-species approach is essential for understanding the site of integration, computational mechanism and functional role of this processing. To improve our knowledge we must rely on brain imaging with improved spatial and temporal resolution and paradigms which can measure internal processes in the human brain, and on the bridging of disciplines in order to characterize this processing at cellular and circuit levels. We highlight apical amplification as one potential mechanism for integrating feedforward and feedback inputs within pyramidal neurons in the rodent brain. We reflect on the challenges and progress in applying this model neuronal process to the study of human cognition. We conclude that cortical-layer specific measures in humans will be an essential contribution for better understanding the landscape of information in cortical feedback, helping to bridge the explanatory gap.
Collapse
Affiliation(s)
- Lucy S Petro
- Centre for Cognitive Neuroimaging, Institute of Neuroscience and Psychology, University of Glasgow, 58 Hillhead Street, Glasgow G12 8QB, Scotland, United Kingdom.
| | - Lars Muckli
- Centre for Cognitive Neuroimaging, Institute of Neuroscience and Psychology, University of Glasgow, 58 Hillhead Street, Glasgow G12 8QB, Scotland, United Kingdom.
| |
Collapse
|
28
|
Marblestone AH, Wayne G, Kording KP. Toward an Integration of Deep Learning and Neuroscience. Front Comput Neurosci 2016; 10:94. [PMID: 27683554 PMCID: PMC5021692 DOI: 10.3389/fncom.2016.00094] [Citation(s) in RCA: 234] [Impact Index Per Article: 29.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2016] [Accepted: 08/24/2016] [Indexed: 01/22/2023] Open
Abstract
Neuroscience has focused on the detailed implementation of computation, studying neural codes, dynamics and circuits. In machine learning, however, artificial neural networks tend to eschew precisely designed codes, dynamics or circuits in favor of brute force optimization of a cost function, often using simple and relatively uniform initial architectures. Two recent developments have emerged within machine learning that create an opportunity to connect these seemingly divergent perspectives. First, structured architectures are used, including dedicated systems for attention, recursion and various forms of short- and long-term memory storage. Second, cost functions and training procedures have become more complex and are varied across layers and over time. Here we think about the brain in terms of these ideas. We hypothesize that (1) the brain optimizes cost functions, (2) the cost functions are diverse and differ across brain locations and over development, and (3) optimization operates within a pre-structured architecture matched to the computational problems posed by behavior. In support of these hypotheses, we argue that a range of implementations of credit assignment through multiple layers of neurons are compatible with our current knowledge of neural circuitry, and that the brain's specialized systems can be interpreted as enabling efficient optimization for specific problem classes. Such a heterogeneously optimized system, enabled by a series of interacting cost functions, serves to make learning data-efficient and precisely targeted to the needs of the organism. We suggest directions by which neuroscience could seek to refine and test these hypotheses.
Collapse
Affiliation(s)
- Adam H. Marblestone
- Synthetic Neurobiology Group, Massachusetts Institute of Technology, Media LabCambridge, MA, USA
| | | | - Konrad P. Kording
- Rehabilitation Institute of Chicago, Northwestern UniversityChicago, IL, USA
| |
Collapse
|
29
|
van der Togt C, Stănişor L, Pooresmaeili A, Albantakis L, Deco G, Roelfsema PR. Learning a New Selection Rule in Visual and Frontal Cortex. Cereb Cortex 2016; 26:3611-26. [PMID: 27269960 PMCID: PMC4961027 DOI: 10.1093/cercor/bhw155] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/01/2023] Open
Abstract
How do you make a decision if you do not know the rules of the game? Models of sensory decision-making suggest that choices are slow if evidence is weak, but they may only apply if the subject knows the task rules. Here, we asked how the learning of a new rule influences neuronal activity in the visual (area V1) and frontal cortex (area FEF) of monkeys. We devised a new icon-selection task. On each day, the monkeys saw 2 new icons (small pictures) and learned which one was relevant. We rewarded eye movements to a saccade target connected to the relevant icon with a curve. Neurons in visual and frontal cortex coded the monkey's choice, because the representation of the selected curve was enhanced. Learning delayed the neuronal selection signals and we uncovered the cause of this delay in V1, where learning to select the relevant icon caused an early suppression of surrounding image elements. These results demonstrate that the learning of a new rule causes a transition from fast and random decisions to a more considerate strategy that takes additional time and they reveal the contribution of visual and frontal cortex to the learning process.
Collapse
Affiliation(s)
- Chris van der Togt
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, An Institute of the Royal Netherlands Academy of Arts and Sciences, 1105 BA Amsterdam, The Netherlands
| | - Liviu Stănişor
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, An Institute of the Royal Netherlands Academy of Arts and Sciences, 1105 BA Amsterdam, The Netherlands
| | - Arezoo Pooresmaeili
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, An Institute of the Royal Netherlands Academy of Arts and Sciences, 1105 BA Amsterdam, The Netherlands
| | - Larissa Albantakis
- Madison School of Medicine, Department of Psychiatry, University of Wisconsin, 6001 Research Park Boulevard, Madison, WI 53719, USA
| | - Gustavo Deco
- Dept. de Tecnologies de la Informació i les Comunicacions, Universitat Pompeu Fabra, C\ Tanger, 122-140, 08018 Barcelona, Spain
| | - Pieter R Roelfsema
- Department of Vision and Cognition, Netherlands Institute for Neuroscience, An Institute of the Royal Netherlands Academy of Arts and Sciences, 1105 BA Amsterdam, The Netherlands Department of Integrative Neurophysiology, Centre for Neurogenomics and Cognitive Research, VU University Amsterdam, Amsterdam, The Netherlands Psychiatry Department, Academic Medical Center, 1105 AZ Amsterdam, The Netherlands
| |
Collapse
|
30
|
Brosch T, Neumann H, Roelfsema PR. Reinforcement Learning of Linking and Tracing Contours in Recurrent Neural Networks. PLoS Comput Biol 2015; 11:e1004489. [PMID: 26496502 PMCID: PMC4619762 DOI: 10.1371/journal.pcbi.1004489] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2015] [Accepted: 08/05/2015] [Indexed: 11/30/2022] Open
Abstract
The processing of a visual stimulus can be subdivided into a number of stages. Upon stimulus presentation there is an early phase of feedforward processing where the visual information is propagated from lower to higher visual areas for the extraction of basic and complex stimulus features. This is followed by a later phase where horizontal connections within areas and feedback connections from higher areas back to lower areas come into play. In this later phase, image elements that are behaviorally relevant are grouped by Gestalt grouping rules and are labeled in the cortex with enhanced neuronal activity (object-based attention in psychology). Recent neurophysiological studies revealed that reward-based learning influences these recurrent grouping processes, but it is not well understood how rewards train recurrent circuits for perceptual organization. This paper examines the mechanisms for reward-based learning of new grouping rules. We derive a learning rule that can explain how rewards influence the information flow through feedforward, horizontal and feedback connections. We illustrate the efficiency with two tasks that have been used to study the neuronal correlates of perceptual organization in early visual cortex. The first task is called contour-integration and demands the integration of collinear contour elements into an elongated curve. We show how reward-based learning causes an enhancement of the representation of the to-be-grouped elements at early levels of a recurrent neural network, just as is observed in the visual cortex of monkeys. The second task is curve-tracing where the aim is to determine the endpoint of an elongated curve composed of connected image elements. If trained with the new learning rule, neural networks learn to propagate enhanced activity over the curve, in accordance with neurophysiological data. We close the paper with a number of model predictions that can be tested in future neurophysiological and computational studies. Our experience with the visual world allows us to group image elements that belong to the same perceptual object and to segregate them from other objects and the background. If subjects learn to group contour elements, this experience influences neuronal activity in early visual cortical areas, including the primary visual cortex (V1). Learning presumably depends on alterations in the pattern of connections within and between areas of the visual cortex. However, the processes that control changes in connectivity are not well understood. Here we present the first computational model that can train a neural network to integrate collinear contour elements into elongated curves and to trace a curve through the visual field. The new learning algorithm trains fully recurrent neural networks, provided the connectivity causes the networks to reach a stable state. The model reproduces the behavioral performance of monkeys trained in these tasks and explains the patterns of neuronal activity in the visual cortex that emerge during learning, which is remarkable because the only feedback for the model is a reward for successful trials. We discuss a number of the model predictions that can be tested in future neuroscientific work.
Collapse
Affiliation(s)
- Tobias Brosch
- University of Ulm, Institute of Neural Information Processing, Ulm, Germany
| | - Heiko Neumann
- University of Ulm, Institute of Neural Information Processing, Ulm, Germany
- * E-mail:
| | - Pieter R. Roelfsema
- Department of Vision & Cognition, Netherlands Institute for Neuroscience (KNAW), Amsterdam, The Netherlands
- Department of Integrative Neurophysiology, Center for Neurogenomics and Cognitive Research, VU University, Amsterdam, The Netherlands
- Psychiatry Department, Academic Medical Center, Amsterdam, The Netherlands
| |
Collapse
|
31
|
Jensen G, Muñoz F, Alkan Y, Ferrera VP, Terrace HS. Implicit Value Updating Explains Transitive Inference Performance: The Betasort Model. PLoS Comput Biol 2015; 11:e1004523. [PMID: 26407227 PMCID: PMC4583549 DOI: 10.1371/journal.pcbi.1004523] [Citation(s) in RCA: 21] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2015] [Accepted: 08/24/2015] [Indexed: 11/19/2022] Open
Abstract
Transitive inference (the ability to infer that B > D given that B > C and C > D) is a widespread characteristic of serial learning, observed in dozens of species. Despite these robust behavioral effects, reinforcement learning models reliant on reward prediction error or associative strength routinely fail to perform these inferences. We propose an algorithm called betasort, inspired by cognitive processes, which performs transitive inference at low computational cost. This is accomplished by (1) representing stimulus positions along a unit span using beta distributions, (2) treating positive and negative feedback asymmetrically, and (3) updating the position of every stimulus during every trial, whether that stimulus was visible or not. Performance was compared for rhesus macaques, humans, and the betasort algorithm, as well as Q-learning, an established reward-prediction error (RPE) model. Of these, only Q-learning failed to respond above chance during critical test trials. Betasort’s success (when compared to RPE models) and its computational efficiency (when compared to full Markov decision process implementations) suggests that the study of reinforcement learning in organisms will be best served by a feature-driven approach to comparing formal models. Although machine learning systems can solve a wide variety of problems, they remain limited in their ability to make logical inferences. We developed a new computational model, called betasort, which addresses these limitations for a certain class of problems: Those in which the algorithm must infer the order of a set of items by trial and error. Unlike extant machine learning systems (but like children and many non-human animals), betasort is able to perform “transitive inferences” about the ordering of a set of images. The patterns of error made by betasort resemble those made by children and non-human animals, and the resulting learning achieved at low computational cost. Additionally, betasort is difficult to classify as either “model-free” or “model-based” according to the formal specifications of those classifications in the machine learning literature. One of the broader implications of these results is that achieving a more comprehensive understanding of how the brain learns will require analysts to entertain other candidate learning models.
Collapse
Affiliation(s)
- Greg Jensen
- Department of Neuroscience, Columbia University, New York, New York, United States of America
- Department of Psychology, Columbia University, New York, New York, United States of America
- * E-mail:
| | - Fabian Muñoz
- Department of Neuroscience, Columbia University, New York, New York, United States of America
| | - Yelda Alkan
- Department of Neuroscience, Columbia University, New York, New York, United States of America
| | - Vincent P. Ferrera
- Department of Neuroscience, Columbia University, New York, New York, United States of America
- Department of Psychiatry, Columbia University, New York, New York, United States of America
| | - Herbert S. Terrace
- Department of Psychology, Columbia University, New York, New York, United States of America
| |
Collapse
|
32
|
Tan AYY. Spatial diversity of spontaneous activity in the cortex. Front Neural Circuits 2015; 9:48. [PMID: 26441547 PMCID: PMC4585302 DOI: 10.3389/fncir.2015.00048] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2015] [Accepted: 08/24/2015] [Indexed: 12/05/2022] Open
Abstract
The neocortex is a layered sheet across which a basic organization is thought to widely apply. The variety of spontaneous activity patterns is similar throughout the cortex, consistent with the notion of a basic cortical organization. However, the basic organization is only an outline which needs adjustments and additions to account for the structural and functional diversity across cortical layers and areas. Such diversity suggests that spontaneous activity is spatially diverse in any particular behavioral state. Accordingly, this review summarizes the laminar and areal diversity in cortical activity during fixation and slow oscillations, and the effects of attention, anesthesia and plasticity on the cortical distribution of spontaneous activity. Among questions that remain open, characterizing the spatial diversity in spontaneous membrane potential may help elucidate how differences in circuitry among cortical regions supports their varied functions. More work is also needed to understand whether cortical spontaneous activity not only reflects cortical circuitry, but also contributes to determining the outcome of plasticity, so that it is itself a factor shaping the functional diversity of the cortex.
Collapse
Affiliation(s)
- Andrew Y Y Tan
- Center for Perceptual Systems and Department of Neuroscience, The University of Texas at Austin Austin, TX, USA
| |
Collapse
|