1
|
Howard MW, Esfahani ZG, Le B, Sederberg PB. Foundations of a temporal RL. ARXIV 2023:arXiv:2302.10163v1. [PMID: 36866224 PMCID: PMC9980275] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Subscribe] [Scholar Register] [Indexed: 03/04/2023]
Abstract
Recent advances in neuroscience and psychology show that the brain has access to timelines of both the past and the future. Spiking across populations of neurons in many regions of the mammalian brain maintains a robust temporal memory, a neural timeline of the recent past. Behavioral results demonstrate that people can estimate an extended temporal model of the future, suggesting that the neural timeline of the past could extend through the present into the future. This paper presents a mathematical framework for learning and expressing relationships between events in continuous time. We assume that the brain has access to a temporal memory in the form of the real Laplace transform of the recent past. Hebbian associations with a diversity of synaptic time scales are formed between the past and the present that record the temporal relationships between events. Knowing the temporal relationships between the past and the present allows one to predict relationships between the present and the future, thus constructing an extended temporal prediction for the future. Both memory for the past and the predicted future are represented as the real Laplace transform, expressed as the firing rate over populations of neurons indexed by different rate constants s. The diversity of synaptic timescales allows for a temporal record over the much larger time scale of trial history. In this framework, temporal credit assignment can be assessed via a Laplace temporal difference. The Laplace temporal difference compares the future that actually follows a stimulus to the future predicted just before the stimulus was observed. This computational framework makes a number of specific neurophysiological predictions and, taken together, could provide the basis for a future iteration of RL that incorporates temporal memory as a fundamental building block.
Collapse
|
2
|
Shi Z, Gu BM, Glasauer S, Meck WH. Beyond Scalar Timing Theory: Integrating Neural Oscillators with Computational Accessibility in Memory. TIMING & TIME PERCEPTION 2022. [DOI: 10.1163/22134468-bja10059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Abstract
Abstract
One of the major challenges for computational models of timing and time perception is to identify a neurobiological plausible implementation that predicts various behavioral properties, including the scalar property and retrospective timing. The available timing models primarily focus on the scalar property and prospective timing, while virtually ignoring the computational accessibility. Here, we first selectively review timing models based on ramping activity, oscillatory pattern, and time cells, and discuss potential challenges for the existing models. We then propose a multifrequency oscillatory model that offers computational accessibility, which could account for a much broader range of timing features, including both retrospective and prospective timing.
Collapse
Affiliation(s)
- Zhuanghua Shi
- Department of Experimental Psychology, Ludwig Maximilian University of Munich, 80802 Munich, Germany
| | - Bon-Mi Gu
- Department of Neurology and Neurological Sciences, Stanford University, Stanford, CA 94305, USA
| | - Stefan Glasauer
- Chair of Computational Neuroscience, Brandenburg University of Technology Cottbus, 03046 Cottbus, Germany
| | - Warren H. Meck
- Department of Psychology and Neuroscience, Duke University, Durham, NC 27708, USA
| |
Collapse
|
3
|
Houser TM. Spatialization of Time in the Entorhinal-Hippocampal System. Front Behav Neurosci 2022; 15:807197. [PMID: 35069143 PMCID: PMC8770534 DOI: 10.3389/fnbeh.2021.807197] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2021] [Accepted: 12/06/2021] [Indexed: 11/19/2022] Open
Abstract
The functional role of the entorhinal-hippocampal system has been a long withstanding mystery. One key theory that has become most popular is that the entorhinal-hippocampal system represents space to facilitate navigation in one's surroundings. In this Perspective article, I introduce a novel idea that undermines the inherent uniqueness of spatial information in favor of time driving entorhinal-hippocampal activity. Specifically, by spatializing events that occur in succession (i.e., across time), the entorhinal-hippocampal system is critical for all types of cognitive representations. I back up this argument with empirical evidence that hints at a role for the entorhinal-hippocampal system in non-spatial representation, and computational models of the logarithmic compression of time in the brain.
Collapse
Affiliation(s)
- Troy M. Houser
- Department of Psychology, University of Oregon, Eugene, OR, United States
| |
Collapse
|
4
|
Mikhael JG, Lai L, Gershman SJ. Rational inattention and tonic dopamine. PLoS Comput Biol 2021; 17:e1008659. [PMID: 33760806 PMCID: PMC7990190 DOI: 10.1371/journal.pcbi.1008659] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2019] [Accepted: 12/28/2020] [Indexed: 11/27/2022] Open
Abstract
Slow-timescale (tonic) changes in dopamine (DA) contribute to a wide variety of processes in reinforcement learning, interval timing, and other domains. Furthermore, changes in tonic DA exert distinct effects depending on when they occur (e.g., during learning vs. performance) and what task the subject is performing (e.g., operant vs. classical conditioning). Two influential theories of tonic DA-the average reward theory and the Bayesian theory in which DA controls precision-have each been successful at explaining a subset of empirical findings. But how the same DA signal performs two seemingly distinct functions without creating crosstalk is not well understood. Here we reconcile the two theories under the unifying framework of 'rational inattention,' which (1) conceptually links average reward and precision, (2) outlines how DA manipulations affect this relationship, and in so doing, (3) captures new empirical phenomena. In brief, rational inattention asserts that agents can increase their precision in a task (and thus improve their performance) by paying a cognitive cost. Crucially, whether this cost is worth paying depends on average reward availability, reported by DA. The monotonic relationship between average reward and precision means that the DA signal contains the information necessary to retrieve the precision. When this information is needed after the task is performed, as presumed by Bayesian inference, acute manipulations of DA will bias behavior in predictable ways. We show how this framework reconciles a remarkably large collection of experimental findings. In reinforcement learning, the rational inattention framework predicts that learning from positive and negative feedback should be enhanced in high and low DA states, respectively, and that DA should tip the exploration-exploitation balance toward exploitation. In interval timing, this framework predicts that DA should increase the speed of the internal clock and decrease the extent of interference by other temporal stimuli during temporal reproduction (the central tendency effect). Finally, rational inattention makes the new predictions that these effects should be critically dependent on the controllability of rewards, that post-reward delays in intertemporal choice tasks should be underestimated, and that average reward manipulations should affect the speed of the clock-thus capturing empirical findings that are unexplained by either theory alone. Our results suggest that a common computational repertoire may underlie the seemingly heterogeneous roles of DA.
Collapse
Affiliation(s)
- John G. Mikhael
- Program in Neuroscience, Harvard Medical School, Boston, Massachusetts, United States of America
- MD-PhD Program, Harvard Medical School, Boston, Massachusetts, United States of America
| | - Lucy Lai
- Program in Neuroscience, Harvard Medical School, Boston, Massachusetts, United States of America
| | - Samuel J. Gershman
- Department of Psychology and Center for Brain Science, Harvard University, Cambridge, Massachusetts, United States of America
| |
Collapse
|
5
|
Hsu A, Marzen SE. Time cells might be optimized for predictive capacity, not redundancy reduction or memory capacity. Phys Rev E 2021; 102:062404. [PMID: 33465990 DOI: 10.1103/physreve.102.062404] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Accepted: 10/08/2020] [Indexed: 11/07/2022]
Abstract
Recently, researchers have found time cells in the hippocampus that appear to contain information about the timing of past events. Some researchers have argued that time cells are taking a Laplace transform of their input in order to reconstruct the past stimulus. We argue that stimulus prediction, not stimulus reconstruction or redundancy reduction, is in better agreement with observed responses of time cells. In the process, we introduce new analyses of nonlinear, continuous-time reservoirs that model these time cells.
Collapse
Affiliation(s)
- Alexander Hsu
- W. M. Keck Science Department, Claremont, California 91711, USA
| | - Sarah E Marzen
- W. M. Keck Science Department, Claremont, California 91711, USA
| |
Collapse
|
6
|
Liu Y, Howard MW. Generation of Scale-Invariant Sequential Activity in Linear Recurrent Networks. Neural Comput 2020; 32:1379-1407. [PMID: 32433902 DOI: 10.1162/neco_a_01288] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Sequential neural activity has been observed in many parts of the brain and has been proposed as a neural mechanism for memory. The natural world expresses temporal relationships at a wide range of scales. Because we cannot know the relevant scales a priori, it is desirable that memory, and thus the generated sequences, is scale invariant. Although recurrent neural network models have been proposed as a mechanism for generating sequences, the requirements for scale-invariant sequences are not known. This letter reports the constraints that enable a linear recurrent neural network model to generate scale-invariant sequential activity. A straightforward eigendecomposition analysis results in two independent conditions that are required for scale invariance for connectivity matrices with real, distinct eigenvalues. First, the eigenvalues of the network must be geometrically spaced. Second, the eigenvectors must be related to one another via translation. These constraints are easily generalizable for matrices that have complex and distinct eigenvalues. Analogous albeit less compact constraints hold for matrices with degenerate eigenvalues. These constraints, along with considerations on initial conditions, provide a general recipe to build linear recurrent neural networks that support scale-invariant sequential activity.
Collapse
Affiliation(s)
- Yue Liu
- Department of Physics and Center for Systems Neuroscience, Boston University, Boston, MA 02215, U.S.A.
| | - Marc W Howard
- Department of Physics, Center for Systems Neuroscience, and Department of Psychological and Brain Sciences, Boston University, Boston, MA 02215, U.S.A.
| |
Collapse
|
7
|
Tiganj Z, Gershman SJ, Sederberg PB, Howard MW. Estimating Scale-Invariant Future in Continuous Time. Neural Comput 2019; 31:681-709. [PMID: 30764739 PMCID: PMC6959535 DOI: 10.1162/neco_a_01171] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Natural learners must compute an estimate of future outcomes that follow from a stimulus in continuous time. Widely used reinforcement learning algorithms discretize continuous time and estimate either transition functions from one step to the next (model-based algorithms) or a scalar value of exponentially discounted future reward using the Bellman equation (model-free algorithms). An important drawback of model-based algorithms is that computational cost grows linearly with the amount of time to be simulated. An important drawback of model-free algorithms is the need to select a timescale required for exponential discounting. We present a computational mechanism, developed based on work in psychology and neuroscience, for computing a scale-invariant timeline of future outcomes. This mechanism efficiently computes an estimate of inputs as a function of future time on a logarithmically compressed scale and can be used to generate a scale-invariant power-law-discounted estimate of expected future reward. The representation of future time retains information about what will happen when. The entire timeline can be constructed in a single parallel operation that generates concrete behavioral and neural predictions. This computational mechanism could be incorporated into future reinforcement learning algorithms.
Collapse
Affiliation(s)
- Zoran Tiganj
- Center for Memory and Brain, Department of Psychological and Brain Sciences, Boston, MA 02215, U.S.A.
| | - Samuel J Gershman
- Department of Psychology and Center for Brain Science, Harvard University, Cambridge, MA 02138, U.S.A.
| | - Per B Sederberg
- Department of Psychology, University of Virginia, Charlottesville, VA, 22904, U.S.A.
| | - Marc W Howard
- Center for Memory and Brain, Department of Psychological and Brain Sciences, Boston, MA 02215, U.S.A.
| |
Collapse
|
8
|
Liu Y, Tiganj Z, Hasselmo ME, Howard MW. A neural microcircuit model for a scalable scale-invariant representation of time. Hippocampus 2019; 29:260-274. [PMID: 30421473 PMCID: PMC7001882 DOI: 10.1002/hipo.22994] [Citation(s) in RCA: 29] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2017] [Revised: 05/21/2018] [Accepted: 05/31/2018] [Indexed: 11/10/2022]
Abstract
Scale-invariant timing has been observed in a wide range of behavioral experiments. The firing properties of recently described time cells provide a possible neural substrate for scale-invariant behavior. Earlier neural circuit models do not produce scale-invariant neural sequences. In this article, we present a biologically detailed network model based on an earlier mathematical algorithm. The simulations incorporate exponentially decaying persistent firing maintained by the calcium-activated nonspecific (CAN) cationic current and a network structure given by the inverse Laplace transform to generate time cells with scale-invariant firing rates. This model provides the first biologically detailed neural circuit for generating scale-invariant time cells. The circuit that implements the inverse Laplace transform merely consists of off-center/on-surround receptive fields. Critically, rescaling temporal sequences can be accomplished simply via cortical gain control (changing the slope of the f-I curve).
Collapse
Affiliation(s)
- Yue Liu
- Department of Physics, Boston University, Boston, MA 02215
- Center for Memory and Brain, Boston University, Boston, MA 02215
- Center for Systems Neuroscience, Boston University, Boston, MA 02215
| | - Zoran Tiganj
- Center for Memory and Brain, Boston University, Boston, MA 02215
- Center for Systems Neuroscience, Boston University, Boston, MA 02215
| | - Michael E. Hasselmo
- Center for Memory and Brain, Boston University, Boston, MA 02215
- Center for Systems Neuroscience, Boston University, Boston, MA 02215
| | - Marc W. Howard
- Department of Physics, Boston University, Boston, MA 02215
- Center for Memory and Brain, Boston University, Boston, MA 02215
- Center for Systems Neuroscience, Boston University, Boston, MA 02215
| |
Collapse
|
9
|
Howard MW, Luzardo A, Tiganj Z. Evidence accumulation in a Laplace domain decision space. COMPUTATIONAL BRAIN & BEHAVIOR 2018; 1:237-251. [PMID: 31131363 PMCID: PMC6530931 DOI: 10.1007/s42113-018-0016-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
Abstract
Evidence accumulation models of simple decision-making have long assumed that the brain estimates a scalar decision variable corresponding to the log-likelihood ratio of the two alternatives. Typical neural implementations of this algorithmic cognitive model assume that large numbers of neurons are each noisy exemplars of the scalar decision variable. Here we propose a neural implementation of the diffusion model in which many neurons construct and maintain the Laplace transform of the distance to each of the decision bounds. As in classic findings from brain regions including LIP, the firing rate of neurons coding for the Laplace transform of net accumulated evidence grows to a bound during random dot motion tasks. However, rather than noisy exemplars of a single mean value, this approach makes the novel prediction that firing rates grow to the bound exponentially; across neurons there should be a distribution of different rates. A second set of neurons records an approximate inversion of the Laplace transform; these neurons directly estimate net accumulated evidence. In analogy to time cells and place cells observed in the hippocampus and other brain regions, the neurons in this second set have receptive fields along a "decision axis." This finding is consistent with recent findings from rodent recordings. This theoretical approach places simple evidence accumulation models in the same mathematical language as recent proposals for representing time and space in cognitive models for memory.
Collapse
Affiliation(s)
- Marc W Howard
- Department of Psychological and Brain Sciences, Department of Physics, Boston University
| | - Andre Luzardo
- Department of Psychological and Brain Sciences, Department of Physics, Boston University
| | - Zoran Tiganj
- Department of Psychological and Brain Sciences, Department of Physics, Boston University
| |
Collapse
|
10
|
Tiganj Z, Cromer JA, Roy JE, Miller EK, Howard MW. Compressed Timeline of Recent Experience in Monkey Lateral Prefrontal Cortex. J Cogn Neurosci 2018; 30:935-950. [PMID: 29698121 PMCID: PMC7004225 DOI: 10.1162/jocn_a_01273] [Citation(s) in RCA: 31] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Cognitive theories suggest that working memory maintains not only the identity of recently presented stimuli but also a sense of the elapsed time since the stimuli were presented. Previous studies of the neural underpinnings of working memory have focused on sustained firing, which can account for maintenance of the stimulus identity, but not for representation of the elapsed time. We analyzed single-unit recordings from the lateral prefrontal cortex of macaque monkeys during performance of a delayed match-to-category task. Each sample stimulus triggered a consistent sequence of neurons, with each neuron in the sequence firing during a circumscribed period. These sequences of neurons encoded both stimulus identity and elapsed time. The encoding of elapsed time became less precise as the sample stimulus receded into the past. These findings suggest that working memory includes a compressed timeline of what happened when, consistent with long-standing cognitive theories of human memory.
Collapse
Affiliation(s)
- Zoran Tiganj
- Center for Memory and Brain, Department of Psychological and Brain Sciences, Boston University, Boston, MA 02215
| | - Jason A. Cromer
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Jefferson E. Roy
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Earl K. Miller
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA 02139
| | - Marc W. Howard
- Center for Memory and Brain, Department of Psychological and Brain Sciences, Boston University, Boston, MA 02215
| |
Collapse
|
11
|
Howard MW. Memory as Perception of the Past: Compressed Time inMind and Brain. Trends Cogn Sci 2018; 22:124-136. [PMID: 29389352 PMCID: PMC5881576 DOI: 10.1016/j.tics.2017.11.004] [Citation(s) in RCA: 40] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2017] [Revised: 11/07/2017] [Accepted: 11/16/2017] [Indexed: 01/27/2023]
Abstract
In the visual system retinal space is compressed such that acuity decreases further from the fovea. Different forms of memory may rely on a compressed representation of time, manifested as decreased accuracy for events that happened further in the past. Neurophysiologically, "time cells" show receptive fields in time. Analogous to the compression of visual space, time cells show less acuity for events further in the past. Behavioral evidence suggests memory can be accessed by scanning a compressed temporal representation, analogous to visual search. This suggests a common computational language for visual attention and memory retrieval. In this view, time functions like a scaffolding that organizes memories in much the same way that retinal space functions like a scaffolding for visual perception.
Collapse
Affiliation(s)
- Marc W Howard
- Center for Memory and Brain, Department of Psychologicaland Brain Sciences, Department of Physics, Boston University, Boston, MA, USA.
| |
Collapse
|