1
|
Yan Y, Hunt LT, Hassall CD. Reward positivity affects temporal interval production in a continuous timing task. Psychophysiology 2024; 61:e14589. [PMID: 38615339 DOI: 10.1111/psyp.14589] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Revised: 02/26/2024] [Accepted: 03/20/2024] [Indexed: 04/16/2024]
Abstract
The neural circuits of reward processing and interval timing (including the perception and production of temporal intervals) are functionally intertwined, suggesting that it might be possible for momentary reward processing to influence subsequent timing behavior. Previous animal and human studies have mainly focused on the effect of reward on interval perception, whereas its impact on interval production is less clear. In this study, we examined whether feedback, as an example of performance-contingent reward, biases interval production. We recorded EEG from 20 participants while they engaged in a continuous drumming task with different realistic tempos (1728 trials per participant). Participants received color-coded feedback after each beat about whether they were correct (on time) or incorrect (early or late). Regression-based EEG analysis was used to unmix the rapid occurrence of a feedback response called the reward positivity (RewP), which is traditionally observed in more slow-paced tasks. Using linear mixed modeling, we found that RewP amplitude predicted timing behavior for the upcoming beat. This performance-biasing effect of the RewP was interpreted as reflecting the impact of fluctuations in reward-related anterior cingulate cortex activity on timing, and the necessity of continuous paradigms to make such observations was highlighted.
Collapse
Affiliation(s)
- Yan Yan
- Department of Psychiatry, University of Oxford, Oxford, UK
- Department of Psychology, Stanford University, Stanford, California, USA
| | - Laurence T Hunt
- Department of Psychiatry, University of Oxford, Oxford, UK
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Cameron D Hassall
- Department of Psychiatry, University of Oxford, Oxford, UK
- Department of Psychology, MacEwan University, Edmonton, Alberta, Canada
| |
Collapse
|
2
|
Biane JS, Ladow MA, Stefanini F, Boddu SP, Fan A, Hassan S, Dundar N, Apodaca-Montano DL, Zhou LZ, Fayner V, Woods NI, Kheirbek MA. Neural dynamics underlying associative learning in the dorsal and ventral hippocampus. Nat Neurosci 2023; 26:798-809. [PMID: 37012382 PMCID: PMC10448873 DOI: 10.1038/s41593-023-01296-6] [Citation(s) in RCA: 21] [Impact Index Per Article: 21.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2021] [Accepted: 03/07/2023] [Indexed: 04/05/2023]
Abstract
Animals associate cues with outcomes and update these associations as new information is presented. This requires the hippocampus, yet how hippocampal neurons track changes in cue-outcome associations remains unclear. Using two-photon calcium imaging, we tracked the same dCA1 and vCA1 neurons across days to determine how responses evolve across phases of odor-outcome learning. Initially, odors elicited robust responses in dCA1, whereas, in vCA1, odor responses primarily emerged after learning and embedded information about the paired outcome. Population activity in both regions rapidly reorganized with learning and then stabilized, storing learned odor representations for days, even after extinction or pairing with a different outcome. Additionally, we found stable, robust signals across CA1 when mice anticipated outcomes under behavioral control but not when mice anticipated an inescapable aversive outcome. These results show how the hippocampus encodes, stores and updates learned associations and illuminates the unique contributions of dorsal and ventral hippocampus.
Collapse
Affiliation(s)
- Jeremy S Biane
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA.
| | - Max A Ladow
- Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA, USA
| | - Fabio Stefanini
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Department of Neuroscience, Columbia University, New York, NY, USA
| | - Sayi P Boddu
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA
| | - Austin Fan
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA
| | - Shazreh Hassan
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA
| | - Naz Dundar
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA
- Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA, USA
| | - Daniel L Apodaca-Montano
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA
| | - Lexi Zichen Zhou
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA
| | - Varya Fayner
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA
| | - Nicholas I Woods
- Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA, USA
| | - Mazen A Kheirbek
- Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, San Francisco, CA, USA.
- Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA, USA.
- Kavli Institute for Fundamental Neuroscience, University of California, San Francisco, San Francisco, CA, USA.
- Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA, USA.
- Center for Integrative Neuroscience, University of California, San Francisco, San Francisco, CA, USA.
| |
Collapse
|
3
|
Young ME, Howatt BC. Resource limitations: A taxonomy. Behav Processes 2023; 206:104823. [PMID: 36682436 DOI: 10.1016/j.beproc.2023.104823] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2022] [Revised: 01/02/2023] [Accepted: 01/17/2023] [Indexed: 01/21/2023]
Abstract
Decision making within the context of resource limitations requires balancing the short-term benefits of obtaining a resource and the long-term consequences of depleting those resources. The present manuscript focuses on four types of tasks that share this tradeoff to develop a taxonomy that will encourage a deeper understanding of the psychological processes at play. The four types considered are foraging, common pool traps, deterioration traps, and a novel designation referred to as resource cliffs. All four will be shown to include two opposite processes - depletion of the resource and its replenishment over time. By considering the unique and shared features of these tasks, a taxonomy of features emerges that can be combined to not only create novel tasks but also to shift the research focus to task features rather than specific tasks. The paper closes with a consideration of current theoretical frameworks previously applied to one or more of these resource-limitation tasks as well as the promise of reinforcement learning as a unifying theory.
Collapse
|
4
|
Tsao A, Yousefzadeh SA, Meck WH, Moser MB, Moser EI. The neural bases for timing of durations. Nat Rev Neurosci 2022; 23:646-665. [PMID: 36097049 DOI: 10.1038/s41583-022-00623-3] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/19/2022] [Indexed: 11/10/2022]
Abstract
Durations are defined by a beginning and an end, and a major distinction is drawn between durations that start in the present and end in the future ('prospective timing') and durations that start in the past and end either in the past or the present ('retrospective timing'). Different psychological processes are thought to be engaged in each of these cases. The former is thought to engage a clock-like mechanism that accurately tracks the continuing passage of time, whereas the latter is thought to engage a reconstructive process that utilizes both temporal and non-temporal information from the memory of past events. We propose that, from a biological perspective, these two forms of duration 'estimation' are supported by computational processes that are both reliant on population state dynamics but are nevertheless distinct. Prospective timing is effectively carried out in a single step where the ongoing dynamics of population activity directly serve as the computation of duration, whereas retrospective timing is carried out in two steps: the initial generation of population state dynamics through the process of event segmentation and the subsequent computation of duration utilizing the memory of those dynamics.
Collapse
Affiliation(s)
- Albert Tsao
- Department of Biology, Stanford University, Stanford, CA, USA.
| | | | - Warren H Meck
- Department of Psychology and Neuroscience, Duke University, Durham, NC, USA
| | - May-Britt Moser
- Centre for Neural Computation, Kavli Institute for Systems Neuroscience, Norwegian University of Science and Technology, Trondheim, Norway
| | - Edvard I Moser
- Centre for Neural Computation, Kavli Institute for Systems Neuroscience, Norwegian University of Science and Technology, Trondheim, Norway.
| |
Collapse
|
5
|
Ponzi A, Wickens J. Ramping activity in the striatum. Front Comput Neurosci 2022; 16:902741. [PMID: 35978564 PMCID: PMC9376361 DOI: 10.3389/fncom.2022.902741] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Accepted: 07/05/2022] [Indexed: 11/13/2022] Open
Abstract
Control of the timing of behavior is thought to require the basal ganglia (BG) and BG pathologies impair performance in timing tasks. Temporal interval discrimination depends on the ramping activity of medium spiny neurons (MSN) in the main BG input structure, the striatum, but the underlying mechanisms driving this activity are unclear. Here, we combine an MSN dynamical network model with an action selection system applied to an interval discrimination task. We find that when network parameters are appropriate for the striatum so that slowly fluctuating marginally stable dynamics are intrinsically generated, up and down ramping populations naturally emerge which enable significantly above chance task performance. We show that emergent population activity is in very good agreement with empirical studies and discuss how MSN network dysfunction in disease may alter temporal perception.
Collapse
Affiliation(s)
- Adam Ponzi
- Institute of Biophysics, Italian National Research Council, Palermo, Italy
- *Correspondence: Adam Ponzi
| | - Jeff Wickens
- Neurobiology Research Unit, Okinawa Institute of Science and Technology (OIST), Okinawa, Japan
| |
Collapse
|
6
|
Yin B, Shi Z, Wang Y, Meck WH. Oscillation/Coincidence-Detection Models of Reward-Related Timing in Corticostriatal Circuits. TIMING & TIME PERCEPTION 2022. [DOI: 10.1163/22134468-bja10057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Abstract
Abstract
The major tenets of beat-frequency/coincidence-detection models of reward-related timing are reviewed in light of recent behavioral and neurobiological findings. This includes the emphasis on a core timing network embedded in the motor system that is comprised of a corticothalamic-basal ganglia circuit. Therein, a central hub provides timing pulses (i.e., predictive signals) to the entire brain, including a set of distributed satellite regions in the cerebellum, cortex, amygdala, and hippocampus that are selectively engaged in timing in a manner that is more dependent upon the specific sensory, behavioral, and contextual requirements of the task. Oscillation/coincidence-detection models also emphasize the importance of a tuned ‘perception’ learning and memory system whereby target durations are detected by striatal networks of medium spiny neurons (MSNs) through the coincidental activation of different neural populations, typically utilizing patterns of oscillatory input from the cortex and thalamus or derivations thereof (e.g., population coding) as a time base. The measure of success of beat-frequency/coincidence-detection accounts, such as the Striatal Beat-Frequency model of reward-related timing (SBF), is their ability to accommodate new experimental findings while maintaining their original framework, thereby making testable experimental predictions concerning diagnosis and treatment of issues related to a variety of dopamine-dependent basal ganglia disorders, including Huntington’s and Parkinson’s disease.
Collapse
Affiliation(s)
- Bin Yin
- Department of Psychology and Neuroscience, Duke University, Durham, NC 27708, USA
- School of Psychology, Fujian Normal University, Fuzhou, 350117, Fujian, China
| | - Zhuanghua Shi
- Department of Psychology, Ludwig Maximilian University of Munich, 80802 Munich, Germany
| | - Yaxin Wang
- School of Psychology, Fujian Normal University, Fuzhou, 350117, Fujian, China
| | - Warren H. Meck
- Department of Psychology and Neuroscience, Duke University, Durham, NC 27708, USA
| |
Collapse
|
7
|
Price BH, Gavornik JP. Efficient Temporal Coding in the Early Visual System: Existing Evidence and Future Directions. Front Comput Neurosci 2022; 16:929348. [PMID: 35874317 PMCID: PMC9298461 DOI: 10.3389/fncom.2022.929348] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2022] [Accepted: 06/13/2022] [Indexed: 01/16/2023] Open
Abstract
While it is universally accepted that the brain makes predictions, there is little agreement about how this is accomplished and under which conditions. Accurate prediction requires neural circuits to learn and store spatiotemporal patterns observed in the natural environment, but it is not obvious how such information should be stored, or encoded. Information theory provides a mathematical formalism that can be used to measure the efficiency and utility of different coding schemes for data transfer and storage. This theory shows that codes become efficient when they remove predictable, redundant spatial and temporal information. Efficient coding has been used to understand retinal computations and may also be relevant to understanding more complicated temporal processing in visual cortex. However, the literature on efficient coding in cortex is varied and can be confusing since the same terms are used to mean different things in different experimental and theoretical contexts. In this work, we attempt to provide a clear summary of the theoretical relationship between efficient coding and temporal prediction, and review evidence that efficient coding principles explain computations in the retina. We then apply the same framework to computations occurring in early visuocortical areas, arguing that data from rodents is largely consistent with the predictions of this model. Finally, we review and respond to criticisms of efficient coding and suggest ways that this theory might be used to design future experiments, with particular focus on understanding the extent to which neural circuits make predictions from efficient representations of environmental statistics.
Collapse
|
8
|
Polti I, Nau M, Kaplan R, van Wassenhove V, Doeller CF. Rapid encoding of task regularities in the human hippocampus guides sensorimotor timing. eLife 2022; 11:79027. [PMID: 36317500 PMCID: PMC9625083 DOI: 10.7554/elife.79027] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Accepted: 10/02/2022] [Indexed: 11/17/2022] Open
Abstract
The brain encodes the statistical regularities of the environment in a task-specific yet flexible and generalizable format. Here, we seek to understand this process by bridging two parallel lines of research, one centered on sensorimotor timing, and the other on cognitive mapping in the hippocampal system. By combining functional magnetic resonance imaging (fMRI) with a fast-paced time-to-contact (TTC) estimation task, we found that the hippocampus signaled behavioral feedback received in each trial as well as performance improvements across trials along with reward-processing regions. Critically, it signaled performance improvements independent from the tested intervals, and its activity accounted for the trial-wise regression-to-the-mean biases in TTC estimation. This is in line with the idea that the hippocampus supports the rapid encoding of temporal context even on short time scales in a behavior-dependent manner. Our results emphasize the central role of the hippocampus in statistical learning and position it at the core of a brain-wide network updating sensorimotor representations in real time for flexible behavior.
Collapse
Affiliation(s)
- Ignacio Polti
- Kavli Institute for Systems Neuroscience, Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, Jebsen Centre for Alzheimer’s Disease, Norwegian University of Science and TechnologyTrondheimNorway,Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany
| | - Matthias Nau
- Kavli Institute for Systems Neuroscience, Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, Jebsen Centre for Alzheimer’s Disease, Norwegian University of Science and TechnologyTrondheimNorway,Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany
| | - Raphael Kaplan
- Kavli Institute for Systems Neuroscience, Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, Jebsen Centre for Alzheimer’s Disease, Norwegian University of Science and TechnologyTrondheimNorway,Department of Basic Psychology, Clinical Psychology, and Psychobiology, Universitat Jaume ICastellón de la PlanaSpain
| | - Virginie van Wassenhove
- CEA DRF/Joliot, NeuroSpin; INSERM, Cognitive Neuroimaging Unit; CNRS, Université Paris-SaclayGif-Sur-YvetteFrance
| | - Christian F Doeller
- Kavli Institute for Systems Neuroscience, Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, Jebsen Centre for Alzheimer’s Disease, Norwegian University of Science and TechnologyTrondheimNorway,Max Planck Institute for Human Cognitive and Brain SciencesLeipzigGermany,Wilhelm Wundt Institute of Psychology, Leipzig UniversityLeipzigGermany
| |
Collapse
|
9
|
Yousefzadeh SA, Youngkin AE, Lusk NA, Wen S, Meck WH. Bidirectional role of microtubule dynamics in the acquisition and maintenance of temporal information in dorsolateral striatum. Neurobiol Learn Mem 2021; 183:107468. [PMID: 34058346 DOI: 10.1016/j.nlm.2021.107468] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2020] [Revised: 05/13/2021] [Accepted: 05/26/2021] [Indexed: 11/25/2022]
Abstract
Accurate and precise timing is crucial for complex and purposeful behaviors, such as foraging for food or playing a musical instrument. The brain is capable of processing temporal information in a coordinated manner, as if it contains an 'internal clock'. Similar to the need for the brain to orient itself in space in order to understand its surroundings, temporal orientation and tracking is an essential component of cognition as well. While there have been multiple models explaining the neural correlates of timing, independent lines of research appear to converge on the conclusion that populations of neurons in the dorsal striatum encode information relating to where a subject is in time relative to an anticipated goal. Similar to other learning processes, acquisition and maintenance of this temporal information is dependent on synaptic plasticity. Microtubules are cytoskeletal proteins that have been implicated in synaptic plasticity mechanisms and therefore are considered key elements in learning and memory. In this study, we investigated the role of microtubule dynamics in temporal learning by local infusions of microtubule stabilizing and destabilizing agents into the dorsolateral striatum. Our results suggested a bidirectional role for microtubules in timing, such that microtubule stabilization improves the maintenance of learned target durations, but impairs the acquisition of a novel duration. On the other hand, microtubule destabilization enhances the acquisition of novel target durations, while compromising the maintenance of previously learned durations. These findings suggest that microtubule dynamics plays an important role in synaptic plasticity mechanisms in the dorsolateral striatum, which in turn modulates temporal learning and time perception.
Collapse
Affiliation(s)
- S Aryana Yousefzadeh
- Department of Psychology and Neuroscience, Duke University, Durham, NC, United States.
| | - Anna E Youngkin
- Department of Psychology and Neuroscience, Duke University, Durham, NC, United States
| | - Nicholas A Lusk
- Department of Psychology and Neuroscience, Duke University, Durham, NC, United States
| | - Shufan Wen
- Department of Biomedical Engineering, Duke University, Durham, NC, United States
| | - Warren H Meck
- Department of Psychology and Neuroscience, Duke University, Durham, NC, United States
| |
Collapse
|
10
|
Bader F, Wiener M. Awareness of errors and feedback in human time estimation. ACTA ACUST UNITED AC 2021; 28:171-177. [PMID: 33858970 PMCID: PMC8054678 DOI: 10.1101/lm.053108.120] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2020] [Accepted: 03/16/2021] [Indexed: 11/26/2022]
Abstract
Behavioral and electrophysiology studies have shown that humans possess a certain self-awareness of their individual timing ability. However, conflicting reports raise concerns about whether humans can discern the direction of their timing error, calling into question the extent of this timing awareness. To understand the depth of this ability, the impact of nondirectional feedback and reinforcement learning on time perception were examined in a unique temporal reproduction paradigm that involved a mixed set of interval durations and the opportunity to repeat every trial immediately after receiving feedback, essentially allowing a “redo.” Within this task, we tested two groups of participants on versions where nondirectional feedback was provided after every response, or not provided at all. Participants in both groups demonstrated reduced central tendency and exhibited significantly greater accuracy in the redo trial temporal estimates, showcasing metacognitive ability, and an inherent capacity to adjust temporal responses despite the lack of directional information or any feedback at all. Additionally, the feedback group also exhibited an increase in the precision of responses on the redo trials, an effect not observed in the no-feedback group, suggesting that feedback may specifically reduce noise when making a temporal estimate. These findings enhance our understanding of timing self-awareness and can provide insight into what may transpire when this is disrupted.
Collapse
Affiliation(s)
- Farah Bader
- Interdisciplinary Program in Neuroscience, George Mason University, Fairfax, Virginia 22032, USA.,Department of Psychology, George Mason University, Fairfax, Virginia 22032, USA
| | - Martin Wiener
- Department of Psychology, George Mason University, Fairfax, Virginia 22032, USA
| |
Collapse
|
11
|
Mikhael JG, Lai L, Gershman SJ. Rational inattention and tonic dopamine. PLoS Comput Biol 2021; 17:e1008659. [PMID: 33760806 PMCID: PMC7990190 DOI: 10.1371/journal.pcbi.1008659] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/15/2019] [Accepted: 12/28/2020] [Indexed: 11/27/2022] Open
Abstract
Slow-timescale (tonic) changes in dopamine (DA) contribute to a wide variety of processes in reinforcement learning, interval timing, and other domains. Furthermore, changes in tonic DA exert distinct effects depending on when they occur (e.g., during learning vs. performance) and what task the subject is performing (e.g., operant vs. classical conditioning). Two influential theories of tonic DA-the average reward theory and the Bayesian theory in which DA controls precision-have each been successful at explaining a subset of empirical findings. But how the same DA signal performs two seemingly distinct functions without creating crosstalk is not well understood. Here we reconcile the two theories under the unifying framework of 'rational inattention,' which (1) conceptually links average reward and precision, (2) outlines how DA manipulations affect this relationship, and in so doing, (3) captures new empirical phenomena. In brief, rational inattention asserts that agents can increase their precision in a task (and thus improve their performance) by paying a cognitive cost. Crucially, whether this cost is worth paying depends on average reward availability, reported by DA. The monotonic relationship between average reward and precision means that the DA signal contains the information necessary to retrieve the precision. When this information is needed after the task is performed, as presumed by Bayesian inference, acute manipulations of DA will bias behavior in predictable ways. We show how this framework reconciles a remarkably large collection of experimental findings. In reinforcement learning, the rational inattention framework predicts that learning from positive and negative feedback should be enhanced in high and low DA states, respectively, and that DA should tip the exploration-exploitation balance toward exploitation. In interval timing, this framework predicts that DA should increase the speed of the internal clock and decrease the extent of interference by other temporal stimuli during temporal reproduction (the central tendency effect). Finally, rational inattention makes the new predictions that these effects should be critically dependent on the controllability of rewards, that post-reward delays in intertemporal choice tasks should be underestimated, and that average reward manipulations should affect the speed of the clock-thus capturing empirical findings that are unexplained by either theory alone. Our results suggest that a common computational repertoire may underlie the seemingly heterogeneous roles of DA.
Collapse
Affiliation(s)
- John G. Mikhael
- Program in Neuroscience, Harvard Medical School, Boston, Massachusetts, United States of America
- MD-PhD Program, Harvard Medical School, Boston, Massachusetts, United States of America
| | - Lucy Lai
- Program in Neuroscience, Harvard Medical School, Boston, Massachusetts, United States of America
| | - Samuel J. Gershman
- Department of Psychology and Center for Brain Science, Harvard University, Cambridge, Massachusetts, United States of America
| |
Collapse
|
12
|
Fung BJ, Sutlief E, Hussain Shuler MG. Dopamine and the interdependency of time perception and reward. Neurosci Biobehav Rev 2021; 125:380-391. [PMID: 33652021 DOI: 10.1016/j.neubiorev.2021.02.030] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2020] [Revised: 02/16/2021] [Accepted: 02/19/2021] [Indexed: 01/14/2023]
Abstract
Time is a fundamental dimension of our perception of the world and is therefore of critical importance to the organization of human behavior. A corpus of work - including recent optogenetic evidence - implicates striatal dopamine as a crucial factor influencing the perception of time. Another stream of literature implicates dopamine in reward and motivation processes. However, these two domains of research have remained largely separated, despite neurobiological overlap and the apothegmatic notion that "time flies when you're having fun". This article constitutes a review of the literature linking time perception and reward, including neurobiological and behavioral studies. Together, these provide compelling support for the idea that time perception and reward processing interact via a common dopaminergic mechanism.
Collapse
Affiliation(s)
- Bowen J Fung
- The Behavioural Insights Team, Suite 3, Level 13/9 Hunter St, Sydney NSW 2000, Australia.
| | - Elissa Sutlief
- The Solomon H. Snyder Department of Neuroscience, The Johns Hopkins University School of Medicine, Woods Basic Science Building Rm914, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Marshall G Hussain Shuler
- The Solomon H. Snyder Department of Neuroscience, The Johns Hopkins University School of Medicine, Woods Basic Science Building Rm914, 725 N. Wolfe Street, Baltimore, MD 21205, USA; Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N Wolfe Street, Baltimore, MD 21205, USA.
| |
Collapse
|
13
|
Abstract
OBJECTIVES Relapse rates in subjects with an alcohol use disorder who have undergone alcohol detoxification are high, and risk factors vary according to the studied population and the context in which withdrawal occurred. Subjects being treated in psychiatric settings require increased monitoring at the moment of detoxification and during follow-up. It is thus important to identify specific risk factors for relapse in such patients. The objective of this study was to determine factors associated with maintenance of abstinence 2 months after alcohol withdrawal (M2) and to characterize factors associated with later relapses 6 months after withdrawal (M6) among those who were abstainers at M2. METHODS We conducted an ancillary study of a specific psychiatric cohort of subjects with an alcohol use disorder who were followed after withdrawal, by analyzing clinical and biological data collected at M2 and M6. RESULTS The specific factors predictive of future relapse were age, intensity of craving, number of standard glasses consumed, psychiatric comorbidity (depression), and employment and family/marital status. Substance use (other than the use of tobacco) decreased the likelihood of abstinence at M2, whereas a depressive state at the time of alcohol withdrawal increased the likelihood of abstinence at M2. Consumption of other substances and a greater intensity of craving at the time of alcohol withdrawal decreased the likelihood of abstinence at M6. CONCLUSIONS The results of this study highlight the importance of identifying craving, multiple substance use, and psychiatric comorbidities (depression) during comprehensive interviews in follow-up after alcohol withdrawal. In caring for patients after alcohol detoxification, priority should be given to factors that have been shown to enhance the beneficial effects of abstinence, such as mood enhancement.
Collapse
|
14
|
Austen JM, Pickering C, Sprengel R, Sanderson DJ. Dissociating Representations of Time and Number in Reinforcement-Rate Learning by Deletion of the GluA1 AMPA Receptor Subunit in Mice. Psychol Sci 2021; 32:204-217. [PMID: 33395376 PMCID: PMC7883001 DOI: 10.1177/0956797620960392] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Theories of learning differ in whether they assume that learning reflects the strength of an association between memories or symbolic encoding of the statistical properties of events. We provide novel evidence for symbolic encoding of informational variables by demonstrating that sensitivity to time and number in learning is dissociable. Whereas responding in normal mice was dependent on reinforcement rate, responding in mice that lacked the GluA1 AMPA receptor subunit was insensitive to reinforcement rate and, instead, dependent on the number of times a cue had been paired with reinforcement. This suggests that GluA1 is necessary for weighting numeric information by temporal information in order to calculate reinforcement rate. Sample sizes per genotype varied between seven and 23 across six experiments and consisted of both male and female mice. The results provide evidence for explicit encoding of variables by animals rather than implicit encoding via variations in associative strength.
Collapse
Affiliation(s)
| | | | - Rolf Sprengel
- Max Planck Institute for Medical Research, Institute for Anatomy and Cell Biology, Heidelberg University
| | | |
Collapse
|
15
|
Zheng C, Yang S, Parra-Ullauri JM, Garcia-Dominguez A, Bencomo N. Reward-Reinforced Generative Adversarial Networks for Multi-Agent Systems. IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE 2021. [DOI: 10.1109/tetci.2021.3082204] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
16
|
Shao L, Yan Y, Liu Z, Ye X, Xia H, Zhu X, Zhang Y, Zhang Z, Chen H, He W, Liu C, Lu M, Huang Y, Ma L, Sun K, Zhou X, Yang G, Lu J, Tian J. Radiologist-like artificial intelligence for grade group prediction of radical prostatectomy for reducing upgrading and downgrading from biopsy. Theranostics 2020; 10:10200-10212. [PMID: 32929343 PMCID: PMC7481433 DOI: 10.7150/thno.48706] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2020] [Accepted: 08/21/2020] [Indexed: 12/13/2022] Open
Abstract
Rationale: To reduce upgrading and downgrading between needle biopsy (NB) and radical prostatectomy (RP) by predicting patient-level Gleason grade groups (GGs) of RP to avoid over- and under-treatment. Methods: In this study, we retrospectively enrolled 575 patients from two medical institutions. All patients received prebiopsy magnetic resonance (MR) examinations, and pathological evaluations of NB and RP were available. A total of 12,708 slices of original male pelvic MR images (T2-weighted sequences with fat suppression, T2WI-FS) containing 5405 slices of prostate tissue, and 2,753 tumor annotations (only T2WI-FS were annotated using RP pathological sections as ground truth) were analyzed for the prediction of patient-level RP GGs. We present a prostate cancer (PCa) framework, PCa-GGNet, that mimics radiologist behavior based on deep reinforcement learning (DRL). We developed and validated it using a multi-center format. Results: Accuracy (ACC) of our model outweighed NB results (0.815 [95% confidence interval (CI): 0.773-0.857] vs. 0.437 [95% CI: 0.335-0.539]). The PCa-GGNet scored higher (kappa value: 0.761) than NB (kappa value: 0.289). Our model significantly reduced the upgrading rate by 27.9% (P < 0.001) and downgrading rate by 6.4% (P = 0.029). Conclusions: DRL using MRI can be applied to the prediction of patient-level RP GGs to reduce upgrading and downgrading from biopsy, potentially improving the clinical benefits of prostate cancer oncologic controls.
Collapse
Affiliation(s)
- Lizhi Shao
- School of Computer Science and Engineering, Southeast University, Nanjing, China
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
| | - Ye Yan
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Zhenyu Liu
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- CAS Center for Excellence in Brain Science and Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100080, China
| | - Xiongjun Ye
- Urology and lithotripsy center, Peking University People's Hospital, Beijing, China
| | - Haizhui Xia
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Xuehua Zhu
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Yuting Zhang
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Zhiying Zhang
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Huiying Chen
- Department of Radiology, Peking University Third Hospital, Beijing, China
| | - Wei He
- Department of Radiology, Peking University Third Hospital, Beijing, China
| | - Cheng Liu
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Min Lu
- Department of Pathology, Peking University Third Hospital, Beijing, China
| | - Yi Huang
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Lulin Ma
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Kai Sun
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, China
| | - Xuezhi Zhou
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, China
| | - Guanyu Yang
- School of Computer Science and Engineering, Southeast University, Nanjing, China
- LIST, Key Laboratory of Computer Network and Information Integration, Southeast University, Ministry of Education, Nanjing, China
| | - Jian Lu
- Department of Urology, Peking University Third Hospital, Beijing, China
| | - Jie Tian
- CAS Key Laboratory of Molecular Imaging, Beijing Key Laboratory of Molecular Imaging, the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
- Engineering Research Center of Molecular and Neuro Imaging of Ministry of Education, School of Life Science and Technology, Xidian University, Xi'an, China
- Beijing Advanced Innovation Center for Big Data-Based Precision Medicine, School of Medicine and Engineering, Beihang University, Beijing, China
- Key Laboratory of Big Data-Based Precision Medicine (Beihang University),Ministry of Industry and Information Technology, Beijing, China
| |
Collapse
|
17
|
Prediction errors bidirectionally bias time perception. Nat Neurosci 2020; 23:1198-1202. [DOI: 10.1038/s41593-020-0698-3] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2019] [Accepted: 07/27/2020] [Indexed: 11/09/2022]
|
18
|
Chen BW, Yang SH, Lo YC, Wang CF, Wang HL, Hsu CY, Kuo YT, Chen JC, Lin SH, Pan HC, Lee SW, Yu X, Qu B, Kuo CH, Chen YY, Lai HY. Enhancement of Hippocampal Spatial Decoding Using a Dynamic Q-Learning Method With a Relative Reward Using Theta Phase Precession. Int J Neural Syst 2020; 30:2050048. [PMID: 32787635 DOI: 10.1142/s0129065720500483] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
Hippocampal place cells and interneurons in mammals have stable place fields and theta phase precession profiles that encode spatial environmental information. Hippocampal CA1 neurons can represent the animal's location and prospective information about the goal location. Reinforcement learning (RL) algorithms such as Q-learning have been used to build the navigation models. However, the traditional Q-learning ([Formula: see text]Q-learning) limits the reward function once the animals arrive at the goal location, leading to unsatisfactory location accuracy and convergence rates. Therefore, we proposed a revised version of the Q-learning algorithm, dynamical Q-learning ([Formula: see text]Q-learning), which assigns the reward function adaptively to improve the decoding performance. Firing rate was the input of the neural network of [Formula: see text]Q-learning and was used to predict the movement direction. On the other hand, phase precession was the input of the reward function to update the weights of [Formula: see text]Q-learning. Trajectory predictions using [Formula: see text]Q- and [Formula: see text]Q-learning were compared by the root mean squared error (RMSE) between the actual and predicted rat trajectories. Using [Formula: see text]Q-learning, significantly higher prediction accuracy and faster convergence rate were obtained compared with [Formula: see text]Q-learning in all cell types. Moreover, combining place cells and interneurons with theta phase precession improved the convergence rate and prediction accuracy. The proposed [Formula: see text]Q-learning algorithm is a quick and more accurate method to perform trajectory reconstruction and prediction.
Collapse
Affiliation(s)
- Bo-Wei Chen
- Department of Biomedical Engineering, National Yang Ming University, No. 155, Section 2, Linong Street, Taipei 11221, Taiwan.,Department of Mechanical Engineering, National Cheng Kung University, No. 1 University Road, Tainan 70101, Taiwan
| | - Shih-Hung Yang
- Department of Mechanical Engineering, National Cheng Kung University, No. 1 University Road, Tainan 70101, Taiwan
| | - Yu-Chun Lo
- The Ph.D. Program for Neural Regenerative Medicine, College of Medical Science and Technology, Taipei Medical University, No. 250 Wu-Xing Street, Taipei 11031, Taiwan
| | - Ching-Fu Wang
- Department of Biomedical Engineering, National Yang Ming University, No. 155, Section 2, Linong Street, Taipei 11221, Taiwan
| | - Han-Lin Wang
- Department of Biomedical Engineering, National Yang Ming University, No. 155, Section 2, Linong Street, Taipei 11221, Taiwan
| | - Chen-Yang Hsu
- Department of Biomedical Engineering, National Yang Ming University, No. 155, Section 2, Linong Street, Taipei 11221, Taiwan
| | - Yun-Ting Kuo
- Department of Biomedical Engineering, National Yang Ming University, No. 155, Section 2, Linong Street, Taipei 11221, Taiwan
| | - Jung-Chen Chen
- Department of Biomedical Engineering, National Yang Ming University, No. 155, Section 2, Linong Street, Taipei 11221, Taiwan
| | - Sheng-Huang Lin
- Department of Neurology, Hualien Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, No. 707, Section 3, Chung Yang Road, Hualien 97002, Taiwan.,Department of Neurology, School of Medicine, Tzu Chi University, No. 701, Section 3, Zhongyang Road, Hualien 97004, Taiwan
| | - Han-Chi Pan
- National Laboratory Animal Center, No. 99, Lane 130, Section 1, Academia Road, Taipei 11571, Taiwan
| | - Sheng-Wei Lee
- Department of Mechanical Engineering, National Cheng Kung University, No. 1 University Road, Tainan 70101, Taiwan
| | - Xiao Yu
- Department of Neurology of the Second Affiliated Hospital, Interdisciplinary Institute of Neuroscience and Technology, Key Laboratory of Medical Neurobiology of Zhejiang Province, Zhejiang University School of Medicine, Hangzhou 310029, P. R. China.,College of Biomedical Engineering and Instrument Science, Key Laboratory for Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, P. R. China
| | - Boyi Qu
- Department of Neurology of the Second Affiliated Hospital, Interdisciplinary Institute of Neuroscience and Technology, Key Laboratory of Medical Neurobiology of Zhejiang Province, Zhejiang University School of Medicine, Hangzhou 310029, P. R. China.,College of Biomedical Engineering and Instrument Science, Key Laboratory for Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, P. R. China
| | - Chao-Hung Kuo
- Department of Biomedical Engineering, National Yang Ming University, No. 155, Section 2, Linong Street, Taipei 11221, Taiwan.,Department of Neurosurgery, Neurological Institute, Taipei Veterans General Hospital, No. 201, Section 2, Shipai Road, Taipei 11217, Taiwan.,Department of Neurological Surgery, University of Washington, No. 1959 NE Pacific Street, Seattle, WA 98195-6470, U.S.A
| | - You-Yin Chen
- Department of Biomedical Engineering, National Yang Ming University, No. 155, Section 2, Linong Street, Taipei 11221, Taiwan.,The Ph.D. Program for Neural Regenerative Medicine, College of Medical Science and Technology, Taipei Medical University, No. 250 Wu-Xing Street, Taipei 11031, Taiwan
| | - Hsin-Yi Lai
- Department of Neurology of the Second Affiliated Hospital, Interdisciplinary Institute of Neuroscience and Technology, Key Laboratory of Medical Neurobiology of Zhejiang Province, Zhejiang University School of Medicine, Hangzhou 310029, P. R. China.,College of Biomedical Engineering and Instrument Science, Key Laboratory for Biomedical Engineering of Ministry of Education, Zhejiang University, Hangzhou 310027, P. R. China
| |
Collapse
|
19
|
Abstract
Express saccades are unusually short latency, visually guided saccadic eye movements. They are most commonly observed when the fixation spot disappears at a consistent, short interval before a target spot appears at a repeated location. The saccade countermanding task includes no fixation-target gap, variable target presentation times, and the requirement to withhold saccades on some trials. These testing conditions should discourage production of express saccades. However, two macaque monkeys performing the saccade countermanding task produced consistent, multimodal distributions of saccadic latencies. These distributions consisted of a longer mode extending from 200 ms to as much as 600 ms after target presentation and another consistently less than 100 ms after target presentation. Simulations revealed that, by varying express saccade production, monkeys could earn more reward. If express saccades were not rewarded, they were rarely produced. The distinct mechanisms producing express and longer saccade latencies were revealed further by the influence of regularities in the duration of the fixation interval preceding target presentation on saccade latency. Temporal expectancy systematically affected the latencies of regular but not of express saccades. This study highlights that cognitive control can integrate information across trials and strategically elicit intermittent very short latency saccades to acquire more reward.NEW & NOTEWORTHY A serendipitous discovery that macaque monkeys produce express saccades under conditions that should discourage them reveals how cognitive control can adapt behavior to maximize reward.
Collapse
Affiliation(s)
- Steven P Errington
- Department of Psychology, Vanderbilt Vision Research Center, Center for Integrative & Cognitive Neuroscience, Vanderbilt Brain Institute, Vanderbilt University, Nashville, Tennessee
| | - Jeffrey D Schall
- Department of Psychology, Vanderbilt Vision Research Center, Center for Integrative & Cognitive Neuroscience, Vanderbilt Brain Institute, Vanderbilt University, Nashville, Tennessee
| |
Collapse
|
20
|
Martel A, Apicella P. Temporal processing in the striatum: Interplay between midbrain dopamine neurons and striatal cholinergic interneurons. Eur J Neurosci 2020; 53:2090-2099. [DOI: 10.1111/ejn.14741] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2020] [Revised: 04/01/2020] [Accepted: 04/02/2020] [Indexed: 11/28/2022]
Affiliation(s)
- Anne‐Caroline Martel
- Institut de Neurosciences de la Timone UMR 7289 Aix Marseille Université, CNRS Marseille France
| | - Paul Apicella
- Institut de Neurosciences de la Timone UMR 7289 Aix Marseille Université, CNRS Marseille France
| |
Collapse
|
21
|
Abstract
Perceiving, maintaining, and using time intervals in working memory are crucial for animals to anticipate or act correctly at the right time in the ever-changing world. Here, we systematically study the underlying neural mechanisms by training recurrent neural networks to perform temporal tasks or complex tasks in combination with spatial information processing and decision making. We found that neural networks perceive time through state evolution along stereotypical trajectories and produce time intervals by scaling evolution speed. Temporal and nontemporal information is jointly coded in a way that facilitates decoding generalizability. We also provided potential sources for the temporal signals observed in nontiming tasks. Our study revealed the computational principles of a number of experimental phenomena and provided several predictions. To maximize future rewards in this ever-changing world, animals must be able to discover the temporal structure of stimuli and then anticipate or act correctly at the right time. How do animals perceive, maintain, and use time intervals ranging from hundreds of milliseconds to multiseconds in working memory? How is temporal information processed concurrently with spatial information and decision making? Why are there strong neuronal temporal signals in tasks in which temporal information is not required? A systematic understanding of the underlying neural mechanisms is still lacking. Here, we addressed these problems using supervised training of recurrent neural network models. We revealed that neural networks perceive elapsed time through state evolution along stereotypical trajectory, maintain time intervals in working memory in the monotonic increase or decrease of the firing rates of interval-tuned neurons, and compare or produce time intervals by scaling state evolution speed. Temporal and nontemporal information is coded in subspaces orthogonal with each other, and the state trajectories with time at different nontemporal information are quasiparallel and isomorphic. Such coding geometry facilitates the decoding generalizability of temporal and nontemporal information across each other. The network structure exhibits multiple feedforward sequences that mutually excite or inhibit depending on whether their preferences of nontemporal information are similar or not. We identified four factors that facilitate strong temporal signals in nontiming tasks, including the anticipation of coming events. Our work discloses fundamental computational principles of temporal processing, and it is supported by and gives predictions to a number of experimental phenomena.
Collapse
|
22
|
Does temporal predictability of tasks influence task choice? PSYCHOLOGICAL RESEARCH 2020; 85:1066-1083. [PMID: 32067103 PMCID: PMC8049906 DOI: 10.1007/s00426-020-01297-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2019] [Accepted: 01/25/2020] [Indexed: 11/18/2022]
Abstract
Task performance improves when the required tasks are predicted by the preceding time intervals, suggesting that participants form time-based task expectancies. In the present study, we pursued the question whether temporal predictability of tasks can also influence task choice. For this purpose, we conducted three experiments using a hybrid task-switching paradigm (with two tasks) combining forced-choice and free-choice trials. Each trial was preceded by either a short (500 ms) or a long (1500 ms) foreperiod. In forced-choice trials, the instructed task was predicted by the length of the foreperiod (Exp. 1A and 1B: 100% foreperiod-task contingencies; Exp. 2: 80% foreperiod-task contingencies). In the remaining trials, participants were free to choose which task to perform. In all three experiments, we found that participants’ task choice was influenced by the foreperiod-task contingencies implemented in forced-choice trials. Specifically, participants were overall biased to choose tasks compatible with these contingencies; these compatible choice rates were larger for the short compared to the long foreperiod. Our findings suggest that learned time-based task expectancies influence subjects’ voluntary task choice and that an initially present task bias toward the “short” task is not always overcome at the long foreperiod. We discuss potential underlying mechanisms against the background of voluntary task switching and interval timing.
Collapse
|
23
|
Yousefzadeh SA, Hesslow G, Shumyatsky GP, Meck WH. Internal Clocks, mGluR7 and Microtubules: A Primer for the Molecular Encoding of Target Durations in Cerebellar Purkinje Cells and Striatal Medium Spiny Neurons. Front Mol Neurosci 2020; 12:321. [PMID: 31998074 PMCID: PMC6965020 DOI: 10.3389/fnmol.2019.00321] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2019] [Accepted: 12/16/2019] [Indexed: 12/16/2022] Open
Abstract
The majority of studies in the field of timing and time perception have generally focused on sub- and supra-second time scales, specific behavioral processes, and/or discrete neuronal circuits. In an attempt to find common elements of interval timing from a broader perspective, we review the literature and highlight the need for cell and molecular studies that can delineate the neural mechanisms underlying temporal processing. Moreover, given the recent attention to the function of microtubule proteins and their potential contributions to learning and memory consolidation/re-consolidation, we propose that these proteins play key roles in coding temporal information in cerebellar Purkinje cells (PCs) and striatal medium spiny neurons (MSNs). The presence of microtubules at relevant neuronal sites, as well as their adaptability, dynamic structure, and longevity, makes them a suitable candidate for neural plasticity at both intra- and inter-cellular levels. As a consequence, microtubules appear capable of maintaining a temporal code or engram and thereby regulate the firing patterns of PCs and MSNs known to be involved in interval timing. This proposed mechanism would control the storage of temporal information triggered by postsynaptic activation of mGluR7. This, in turn, leads to alterations in microtubule dynamics through a "read-write" memory process involving alterations in microtubule dynamics and their hexagonal lattice structures involved in the molecular basis of temporal memory.
Collapse
Affiliation(s)
- S. Aryana Yousefzadeh
- Department of Psychology and Neuroscience, Duke University, Durham, NC, United States
| | - Germund Hesslow
- Department of Experimental Medical Science, Lund University, Lund, Sweden
| | - Gleb P. Shumyatsky
- Department of Genetics, Rutgers University, Piscataway, NJ, United States
| | - Warren H. Meck
- Department of Psychology and Neuroscience, Duke University, Durham, NC, United States
| |
Collapse
|
24
|
Abstract
Midbrain dopamine signals are widely thought to report reward prediction errors that drive learning in the basal ganglia. However, dopamine has also been implicated in various probabilistic computations, such as encoding uncertainty and controlling exploration. Here, we show how these different facets of dopamine signalling can be brought together under a common reinforcement learning framework. The key idea is that multiple sources of uncertainty impinge on reinforcement learning computations: uncertainty about the state of the environment, the parameters of the value function and the optimal action policy. Each of these sources plays a distinct role in the prefrontal cortex-basal ganglia circuit for reinforcement learning and is ultimately reflected in dopamine activity. The view that dopamine plays a central role in the encoding and updating of beliefs brings the classical prediction error theory into alignment with more recent theories of Bayesian reinforcement learning.
Collapse
Affiliation(s)
- Samuel J Gershman
- Department of Psychology, Center for Brain Science, Harvard University, Cambridge, MA, USA.
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA, USA
| |
Collapse
|
25
|
Mikhael JG, Gershman SJ. Adapting the flow of time with dopamine. J Neurophysiol 2019; 121:1748-1760. [PMID: 30864882 PMCID: PMC6589719 DOI: 10.1152/jn.00817.2018] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/04/2018] [Revised: 02/04/2019] [Accepted: 02/20/2019] [Indexed: 01/25/2023] Open
Abstract
The modulation of interval timing by dopamine (DA) has been well established over decades of research. The nature of this modulation, however, has remained controversial: Although the pharmacological evidence has largely suggested that time intervals are overestimated with higher DA levels, more recent optogenetic work has shown the opposite effect. In addition, a large body of work has asserted DA's role as a "reward prediction error" (RPE), or a teaching signal that allows the basal ganglia to learn to predict future rewards in reinforcement learning tasks. Whether these two seemingly disparate accounts of DA may be related has remained an open question. By taking a reinforcement learning-based approach to interval timing, we show here that the RPE interpretation of DA naturally extends to its role as a modulator of timekeeping and furthermore that this view reconciles the seemingly conflicting observations. We derive a biologically plausible, DA-dependent plasticity rule that can modulate the rate of timekeeping in either direction and whose effect depends on the timing of the DA signal itself. This bidirectional update rule can account for the results from pharmacology and optogenetics as well as the behavioral effects of reward rate on interval timing and the temporal selectivity of striatal neurons. Hence, by adopting a single RPE interpretation of DA, our results take a step toward unifying computational theories of reinforcement learning and interval timing. NEW & NOTEWORTHY How does dopamine (DA) influence interval timing? A large body of pharmacological evidence has suggested that DA accelerates timekeeping mechanisms. However, recent optogenetic work has shown exactly the opposite effect. In this article, we relate DA's role in timekeeping to its most established role, as a critical component of reinforcement learning. This allows us to derive a neurobiologically plausible framework that reconciles a large body of DA's temporal effects, including pharmacological, behavioral, electrophysiological, and optogenetic.
Collapse
Affiliation(s)
- John G Mikhael
- Program in Neuroscience and MD-PhD Program, Harvard Medical School , Boston, Massachusetts
| | - Samuel J Gershman
- Center for Brain Science and Department of Psychology, Harvard University , Cambridge, Massachusetts
| |
Collapse
|
26
|
Zeki M, Balcı F. A Simplified Model of Communication Between Time Cells: Accounting for the Linearly Increasing Timing Imprecision. Front Comput Neurosci 2019; 12:111. [PMID: 30760994 PMCID: PMC6361830 DOI: 10.3389/fncom.2018.00111] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2018] [Accepted: 12/31/2018] [Indexed: 11/13/2022] Open
Abstract
Many organisms can time intervals flexibly on average with high accuracy but substantial variability between the trials. One of the core psychophysical features of interval timing functions relates to the signatures of this timing variability; for a given individual, the standard deviation of timed responses/time estimates is nearly proportional to their central tendency (scalar property). Many studies have aimed at elucidating the neural basis of interval timing based on the neurocomputational principles in a fashion that would explain the scalar property. Recent experimental evidence shows that there is indeed a specialized neural system for timekeeping. This system, referred to as the "time cells," is composed of a group of neurons that fire sequentially as a function of elapsed time. Importantly, the time interval between consecutively firing time cell ensembles has been shown to increase with more elapsed time. However, when the subjective time is calculated by adding the distributions of time intervals between these sequentially firing time cell ensembles, the standard deviation would be compressed by the square root function. In light of this information the question becomes, "How should the signaling between the sequentially firing time cell ensembles be for the resulting variability to increase linearly with time as required by the scalar property?" We developed a simplified model of time cells that offers a mechanism for the synaptic communication of the sequentially firing neurons to address this ubiquitous property of interval timing. The model is composed of a single layer of time cells formulated in the form of integrate-and-fire neurons with feed-forward excitatory connections. The resulting behavior is simple neural wave activity. When this model is simulated with noisy conductances, the standard deviation of the time cell spike times increases proportionally to the mean of the spike-times. We demonstrate that this statistical property of the model outcomes is robustly observed even when the values of the key model parameters are varied.
Collapse
Affiliation(s)
- Mustafa Zeki
- Department of Mathematics, College of Engineering and Technology, American University of the Middle East, Egaila, Kuwait
| | - Fuat Balcı
- Department of Psychology, Koç University, Istanbul, Turkey
| |
Collapse
|
27
|
Temporal updating, behavioral learning, and the phenomenology of time-consciousness. Behav Brain Sci 2019; 42:e254. [DOI: 10.1017/s0140525x19000517] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Abstract
Hoerl & McCormack claim that the temporal updating system only represents the world as present. This generates puzzles regarding the phenomenology of temporal experience. We argue that recent models of reinforcement learning suggest that temporal updating must have a minimal temporal structure; and we suggest that this helps to clarify what it means to experience the world as temporally structured.
Collapse
|
28
|
De Corte BJ, Della Valle RR, Matell MS. Recalibrating timing behavior via expected covariance between temporal cues. eLife 2018; 7:e38790. [PMID: 30387710 PMCID: PMC6235573 DOI: 10.7554/elife.38790] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2018] [Accepted: 10/31/2018] [Indexed: 02/06/2023] Open
Abstract
Individuals must predict future events to proactively guide their behavior. Predicting when events will occur is a critical component of these expectations. Temporal expectations are often generated based on individual cue-duration relationships. However, the durations associated with different environmental cues will often co-vary due to a common cause. We show that timing behavior may be calibrated based on this expected covariance, which we refer to as the 'common cause hypothesis'. In five experiments using rats, we found that when the duration associated with one temporal cue changes, timed-responding to other cues shift in the same direction. Furthermore, training subjects that expecting covariance is not appropriate in a given situation blocks this effect. Finally, we confirmed that this transfer is context-dependent. These results reveal a novel principle that modulates timing behavior, which we predict will apply across a variety of magnitude-expectations.
Collapse
Affiliation(s)
| | - Rebecca R Della Valle
- Department of Psychological and Brain SciencesThe University of DelawareNewarkUnited states
| | - Matthew S Matell
- Department of Psychological and Brain SciencesVillanova UniversityVillanovaUnited states
| |
Collapse
|
29
|
|
30
|
|