1
|
Giannone F, Ebrahimi C, Endrass T, Hansson AC, Schlagenhauf F, Sommer WH. Bad habits-good goals? Meta-analysis and translation of the habit construct to alcoholism. Transl Psychiatry 2024; 14:298. [PMID: 39030169 PMCID: PMC11271507 DOI: 10.1038/s41398-024-02965-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/08/2023] [Revised: 05/19/2024] [Accepted: 05/24/2024] [Indexed: 07/21/2024] Open
Abstract
Excessive alcohol consumption remains a global public health crisis, with millions suffering from alcohol use disorder (AUD, or simply "alcoholism"), leading to significantly reduced life expectancy. This review examines the interplay between habitual and goal-directed behaviors and the associated neurobiological changes induced by chronic alcohol exposure. Contrary to a strict habit-goal dichotomy, our meta-analysis of the published animal experiments combined with a review of human studies reveals a nuanced transition between these behavioral control systems, emphasizing the need for refined terminology to capture the probabilistic nature of decision biases in individuals with a history of chronic alcohol exposure. Furthermore, we distinguish habitual responding from compulsivity, viewing them as separate entities with diverse roles throughout the stages of the addiction cycle. By addressing species-specific differences and translational challenges in habit research, we provide insights to enhance future investigations and inform strategies for combatting AUD.
Collapse
Affiliation(s)
- F Giannone
- Institute of Psychopharmacology, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, 68159, Mannheim, Germany
| | - C Ebrahimi
- Faculty of Psychology, Institute of Clinical Psychology and Psychotherapy, Technische Universität Dresden, 01062, Dresden, Germany
| | - T Endrass
- Faculty of Psychology, Institute of Clinical Psychology and Psychotherapy, Technische Universität Dresden, 01062, Dresden, Germany
| | - A C Hansson
- Institute of Psychopharmacology, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, 68159, Mannheim, Germany
| | - F Schlagenhauf
- Department of Psychotherapy, Campus Charité Mitte, Charité Universitätsmedizin Berlin & St. Hedwig Hospital, 10117, Berlin, Germany
| | - W H Sommer
- Institute of Psychopharmacology, Central Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University, 68159, Mannheim, Germany.
- Bethania Hospital for Psychiatry, Psychosomatics and Psychotherapy, Greifswald, Germany.
- German Center for Mental Health (DZPG), Partner Site Mannheim-Heidelberg-Ulm, 68159, Mannheim, Germany.
| |
Collapse
|
2
|
Attaallah B, Petitet P, Zambellas R, Toniolo S, Maio MR, Ganse-Dumrath A, Irani SR, Manohar SG, Husain M. The role of the human hippocampus in decision-making under uncertainty. Nat Hum Behav 2024; 8:1366-1382. [PMID: 38684870 PMCID: PMC11272595 DOI: 10.1038/s41562-024-01855-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Accepted: 02/29/2024] [Indexed: 05/02/2024]
Abstract
The role of the hippocampus in decision-making is beginning to be more understood. Because of its prospective and inferential functions, we hypothesized that it might be required specifically when decisions involve the evaluation of uncertain values. A group of individuals with autoimmune limbic encephalitis-a condition known to focally affect the hippocampus-were tested on how they evaluate reward against uncertainty compared to reward against another key attribute: physical effort. Across four experiments requiring participants to make trade-offs between reward, uncertainty and effort, patients with acute limbic encephalitis demonstrated blunted sensitivity to reward and effort whenever uncertainty was considered, despite demonstrating intact uncertainty sensitivity. By contrast, the valuation of these two attributes (reward and effort) was intact on uncertainty-free tasks. Reduced sensitivity to changes in reward under uncertainty correlated with the severity of hippocampal damage. Together, these findings provide evidence for a context-sensitive role of the hippocampus in value-based decision-making, apparent specifically under conditions of uncertainty.
Collapse
Affiliation(s)
- Bahaaeddin Attaallah
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK.
| | - Pierre Petitet
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Rhea Zambellas
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Sofia Toniolo
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Maria Raquel Maio
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Akke Ganse-Dumrath
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Sarosh R Irani
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
| | - Sanjay G Manohar
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Masud Husain
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, UK
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| |
Collapse
|
3
|
Aquino TG, Courellis H, Mamelak AN, Rutishauser U, O Doherty JP. Encoding of Predictive Associations in Human Prefrontal and Medial Temporal Neurons During Pavlovian Appetitive Conditioning. J Neurosci 2024; 44:e1628232024. [PMID: 38423764 PMCID: PMC11044193 DOI: 10.1523/jneurosci.1628-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Revised: 01/29/2024] [Accepted: 02/19/2024] [Indexed: 03/02/2024] Open
Abstract
Pavlovian conditioning is thought to involve the formation of learned associations between stimuli and values, and between stimuli and specific features of outcomes. Here, we leveraged human single neuron recordings in ventromedial prefrontal, dorsomedial frontal, hippocampus, and amygdala while patients of both sexes performed an appetitive Pavlovian conditioning task probing both stimulus-value and stimulus-stimulus associations. Ventromedial prefrontal cortex encoded predictive value along with the amygdala, and also encoded predictions about the identity of stimuli that would subsequently be presented, suggesting a role for neurons in this region in encoding predictive information beyond value. Unsigned error signals were found in dorsomedial frontal areas and hippocampus, potentially supporting learning of non-value related outcome features. Our findings implicate distinct human prefrontal and medial temporal neuronal populations in mediating predictive associations which could partially support model-based mechanisms during Pavlovian conditioning.
Collapse
Affiliation(s)
- Tomas G Aquino
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, California 90048
- Computation and Neural Systems, Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California 91125
| | - Hristos Courellis
- Biological Engineering, Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California 91125
| | - Adam N Mamelak
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, California 90048
| | - Ueli Rutishauser
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, California 90048
- Computation and Neural Systems, Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California 91125
| | - John P O Doherty
- Computation and Neural Systems, Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California 91125
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, California 91125
| |
Collapse
|
4
|
Kay K, Biderman N, Khajeh R, Beiran M, Cueva CJ, Shohamy D, Jensen G, Wei XX, Ferrera VP, Abbott LF. Emergent neural dynamics and geometry for generalization in a transitive inference task. PLoS Comput Biol 2024; 20:e1011954. [PMID: 38662797 PMCID: PMC11125559 DOI: 10.1371/journal.pcbi.1011954] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Revised: 05/24/2024] [Accepted: 02/28/2024] [Indexed: 05/25/2024] Open
Abstract
Relational cognition-the ability to infer relationships that generalize to novel combinations of objects-is fundamental to human and animal intelligence. Despite this importance, it remains unclear how relational cognition is implemented in the brain due in part to a lack of hypotheses and predictions at the levels of collective neural activity and behavior. Here we discovered, analyzed, and experimentally tested neural networks (NNs) that perform transitive inference (TI), a classic relational task (if A > B and B > C, then A > C). We found NNs that (i) generalized perfectly, despite lacking overt transitive structure prior to training, (ii) generalized when the task required working memory (WM), a capacity thought to be essential to inference in the brain, (iii) emergently expressed behaviors long observed in living subjects, in addition to a novel order-dependent behavior, and (iv) expressed different task solutions yielding alternative behavioral and neural predictions. Further, in a large-scale experiment, we found that human subjects performing WM-based TI showed behavior inconsistent with a class of NNs that characteristically expressed an intuitive task solution. These findings provide neural insights into a classical relational ability, with wider implications for how the brain realizes relational cognition.
Collapse
Affiliation(s)
- Kenneth Kay
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
- Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
| | - Natalie Biderman
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Psychology, Columbia University, New York, New York, United States of America
| | - Ramin Khajeh
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
| | - Manuel Beiran
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
| | - Christopher J. Cueva
- Department of Brain and Cognitive Sciences, MIT, Cambridge, Massachusetts, United States of America
| | - Daphna Shohamy
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Psychology, Columbia University, New York, New York, United States of America
- The Kavli Institute for Brain Science, Columbia University, New York, New York, United States of America
| | - Greg Jensen
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
- Department of Psychology at Reed College, Portland, Oregon, United States of America
| | - Xue-Xin Wei
- Departments of Neuroscience and Psychology, The University of Texas at Austin, Austin, Texas, United States of America
| | - Vincent P. Ferrera
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
- Department of Psychiatry, Columbia University Medical Center, New York, New York, United States of America
| | - LF Abbott
- Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
- Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
- The Kavli Institute for Brain Science, Columbia University, New York, New York, United States of America
- Department of Neuroscience, Columbia University Medical Center, New York, New York, United States of America
| |
Collapse
|
5
|
Varga V, Petersen P, Zutshi I, Huszar R, Zhang Y, Buzsáki G. Working memory features are embedded in hippocampal place fields. Cell Rep 2024; 43:113807. [PMID: 38401118 PMCID: PMC11044127 DOI: 10.1016/j.celrep.2024.113807] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 12/13/2023] [Accepted: 01/31/2024] [Indexed: 02/26/2024] Open
Abstract
Hippocampal principal neurons display both spatial tuning properties and memory features. Whether this distinction corresponds to separate neuron types or a context-dependent continuum has been debated. We report here that the task-context ("splitter") feature is highly variable along both trial and spatial position axes. Neurons acquire or lose splitter features across trials even when place field features remain unaltered. Multiple place fields of the same neuron can individually encode both past or future run trajectories, implying that splitter fields are under the control of assembly activity. Place fields can be differentiated into subfields by the behavioral choice of the animal, and splitting within subfields evolves across trials. Interneurons also differentiate choices by integrating inputs from pyramidal cells. Finally, bilateral optogenetic inactivation of the medial entorhinal cortex reversibly decreases the fraction of splitter fields. Our findings suggest that place or splitter features are different manifestations of the same hippocampal computation.
Collapse
Affiliation(s)
- Viktor Varga
- Neuroscience Institute, Langone Health, New York University, New York, NY, USA; Subcortical Modulation Research Group, Institute of Experimental Medicine - Hungarian Research Network, Budapest, Hungary
| | - Peter Petersen
- Neuroscience Institute, Langone Health, New York University, New York, NY, USA; Department of Neuroscience, University of Copenhagen, Copenhagen, Denmark
| | - Ipshita Zutshi
- Neuroscience Institute, Langone Health, New York University, New York, NY, USA
| | - Roman Huszar
- Neuroscience Institute, Langone Health, New York University, New York, NY, USA
| | - Yiyao Zhang
- Neuroscience Institute, Langone Health, New York University, New York, NY, USA
| | - György Buzsáki
- Neuroscience Institute, Langone Health, New York University, New York, NY, USA; Department of Neuroscience and Physiology, Langone Health, New York University, New York, NY, USA; Department of Neurology, Langone Health, New York University, New York, NY, USA.
| |
Collapse
|
6
|
Venditto SJC, Miller KJ, Brody CD, Daw ND. Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.28.582617. [PMID: 38464244 PMCID: PMC10925334 DOI: 10.1101/2024.02.28.582617] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
Different brain systems have been hypothesized to subserve multiple "experts" that compete to generate behavior. In reinforcement learning, two general processes, one model-free (MF) and one model-based (MB), are often modeled as a mixture of agents (MoA) and hypothesized to capture differences between automaticity vs. deliberation. However, shifts in strategy cannot be captured by a static MoA. To investigate such dynamics, we present the mixture-of-agents hidden Markov model (MoA-HMM), which simultaneously learns inferred action values from a set of agents and the temporal dynamics of underlying "hidden" states that capture shifts in agent contributions over time. Applying this model to a multi-step,reward-guided task in rats reveals a progression of within-session strategies: a shift from initial MB exploration to MB exploitation, and finally to reduced engagement. The inferred states predict changes in both response time and OFC neural encoding during the task, suggesting that these states are capturing real shifts in dynamics.
Collapse
|
7
|
Maggi S, Hock RM, O'Neill M, Buckley M, Moran PM, Bast T, Sami M, Humphries MD. Tracking subjects' strategies in behavioural choice experiments at trial resolution. eLife 2024; 13:e86491. [PMID: 38426402 PMCID: PMC10959529 DOI: 10.7554/elife.86491] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2023] [Accepted: 02/23/2024] [Indexed: 03/02/2024] Open
Abstract
Investigating how, when, and what subjects learn during decision-making tasks requires tracking their choice strategies on a trial-by-trial basis. Here, we present a simple but effective probabilistic approach to tracking choice strategies at trial resolution using Bayesian evidence accumulation. We show this approach identifies both successful learning and the exploratory strategies used in decision tasks performed by humans, non-human primates, rats, and synthetic agents. Both when subjects learn and when rules change the exploratory strategies of win-stay and lose-shift, often considered complementary, are consistently used independently. Indeed, we find the use of lose-shift is strong evidence that subjects have latently learnt the salient features of a new rewarded rule. Our approach can be extended to any discrete choice strategy, and its low computational cost is ideally suited for real-time analysis and closed-loop control.
Collapse
Affiliation(s)
- Silvia Maggi
- School of Psychology, University of NottinghamNottinghamUnited Kingdom
| | - Rebecca M Hock
- School of Psychology, University of NottinghamNottinghamUnited Kingdom
| | - Martin O'Neill
- School of Psychology, University of NottinghamNottinghamUnited Kingdom
- Department of Health & Nutritional Sciences, Atlantic Technological UniversitySligoIreland
| | - Mark Buckley
- Department of Experimental Psychology, University of OxfordOxfordUnited Kingdom
| | - Paula M Moran
- School of Psychology, University of NottinghamNottinghamUnited Kingdom
- Department of Neuroscience, University of NottinghamNottinghamUnited Kingdom
| | - Tobias Bast
- School of Psychology, University of NottinghamNottinghamUnited Kingdom
- Department of Neuroscience, University of NottinghamNottinghamUnited Kingdom
| | - Musa Sami
- Institute of Mental Health, University of NottinghamNottinghamUnited Kingdom
| | - Mark D Humphries
- School of Psychology, University of NottinghamNottinghamUnited Kingdom
| |
Collapse
|
8
|
Colas JT, O’Doherty JP, Grafton ST. Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts. PLoS Comput Biol 2024; 20:e1011950. [PMID: 38552190 PMCID: PMC10980507 DOI: 10.1371/journal.pcbi.1011950] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Accepted: 02/26/2024] [Indexed: 04/01/2024] Open
Abstract
Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.
Collapse
Affiliation(s)
- Jaron T. Colas
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, California, United States of America
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, California, United States of America
| | - John P. O’Doherty
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, California, United States of America
| | - Scott T. Grafton
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, California, United States of America
| |
Collapse
|
9
|
Blanco-Pozo M, Akam T, Walton ME. Dopamine-independent effect of rewards on choices through hidden-state inference. Nat Neurosci 2024; 27:286-297. [PMID: 38216649 PMCID: PMC10849965 DOI: 10.1038/s41593-023-01542-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2021] [Accepted: 12/01/2023] [Indexed: 01/14/2024]
Abstract
Dopamine is implicated in adaptive behavior through reward prediction error (RPE) signals that update value estimates. There is also accumulating evidence that animals in structured environments can use inference processes to facilitate behavioral flexibility. However, it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine activity, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model where cortex learned to track hidden task states by predicting observations, while basal ganglia learned values and actions via RPEs. This shows that the influence of rewards on choices can stem from dopamine-independent information they convey about the world's state, not the dopaminergic RPEs they produce.
Collapse
Affiliation(s)
- Marta Blanco-Pozo
- Department of Experimental Psychology, Oxford University, Oxford, UK.
- Wellcome Centre for Integrative Neuroimaging, Oxford University, Oxford, UK.
| | - Thomas Akam
- Department of Experimental Psychology, Oxford University, Oxford, UK.
- Wellcome Centre for Integrative Neuroimaging, Oxford University, Oxford, UK.
| | - Mark E Walton
- Department of Experimental Psychology, Oxford University, Oxford, UK.
- Wellcome Centre for Integrative Neuroimaging, Oxford University, Oxford, UK.
| |
Collapse
|
10
|
Yang L, Jin F, Yang L, Li J, Li Z, Li M, Shang Z. The Hippocampus in Pigeons Contributes to the Model-Based Valuation and the Relationship between Temporal Context States. Animals (Basel) 2024; 14:431. [PMID: 38338074 PMCID: PMC10854895 DOI: 10.3390/ani14030431] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/30/2023] [Revised: 01/25/2024] [Accepted: 01/25/2024] [Indexed: 02/12/2024] Open
Abstract
Model-based decision-making guides organism behavior by the representation of the relationships between different states. Previous studies have shown that the mammalian hippocampus (Hp) plays a key role in learning the structure of relationships among experiences. However, the hippocampal neural mechanisms of birds for model-based learning have rarely been reported. Here, we trained six pigeons to perform a two-step task and explore whether their Hp contributes to model-based learning. Behavioral performance and hippocampal multi-channel local field potentials (LFPs) were recorded during the task. We estimated the subjective values using a reinforcement learning model dynamically fitted to the pigeon's choice of behavior. The results show that the model-based learner can capture the behavioral choices of pigeons well throughout the learning process. Neural analysis indicated that high-frequency (12-100 Hz) power in Hp represented the temporal context states. Moreover, dynamic correlation and decoding results provided further support for the high-frequency dependence of model-based valuations. In addition, we observed a significant increase in hippocampal neural similarity at the low-frequency band (1-12 Hz) for common temporal context states after learning. Overall, our findings suggest that pigeons use model-based inferences to learn multi-step tasks, and multiple LFP frequency bands collaboratively contribute to model-based learning. Specifically, the high-frequency (12-100 Hz) oscillations represent model-based valuations, while the low-frequency (1-12 Hz) neural similarity is influenced by the relationship between temporal context states. These results contribute to our understanding of the neural mechanisms underlying model-based learning and broaden the scope of hippocampal contributions to avian behavior.
Collapse
Affiliation(s)
- Lifang Yang
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Fuli Jin
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Long Yang
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Jiajia Li
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Zhihui Li
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
- Institute of Medical Engineering Technology and Data Mining, Zhengzhou University, Zhengzhou 450001, China
| | - Mengmeng Li
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
| | - Zhigang Shang
- School of Electrical and Information Engineering, Zhengzhou University, Zhengzhou 450001, China; (L.Y.); (F.J.); (L.Y.); (J.L.); (Z.L.)
- Henan Key Laboratory of Brain Science and Brain-Computer Interface Technology, Zhengzhou 450001, China
- Institute of Medical Engineering Technology and Data Mining, Zhengzhou University, Zhengzhou 450001, China
| |
Collapse
|
11
|
Mah A, Schiereck SS, Bossio V, Constantinople CM. Distinct value computations support rapid sequential decisions. Nat Commun 2023; 14:7573. [PMID: 37989741 PMCID: PMC10663503 DOI: 10.1038/s41467-023-43250-x] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2023] [Accepted: 11/03/2023] [Indexed: 11/23/2023] Open
Abstract
The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or cache values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.
Collapse
Affiliation(s)
- Andrew Mah
- Center for Neural Science, New York University, New York, NY, 10003, USA
| | | | - Veronica Bossio
- Center for Neural Science, New York University, New York, NY, 10003, USA
- Zuckerman Institute, Columbia University, New York, NY, 10027, USA
| | | |
Collapse
|
12
|
Qü AJ, Tai LH, Hall CD, Tu EM, Eckstein MK, Mishchanchuk K, Lin WC, Chase JB, MacAskill AF, Collins AG, Gershman SJ, Wilbrecht L. Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.10.566306. [PMID: 38014354 PMCID: PMC10680647 DOI: 10.1101/2023.11.10.566306] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
Dopamine release in the nucleus accumbens has been hypothesized to signal reward prediction error, the difference between observed and predicted reward, suggesting a biological implementation for reinforcement learning. Rigorous tests of this hypothesis require assumptions about how the brain maps sensory signals to reward predictions, yet this mapping is still poorly understood. In particular, the mapping is non-trivial when sensory signals provide ambiguous information about the hidden state of the environment. Previous work using classical conditioning tasks has suggested that reward predictions are generated conditional on probabilistic beliefs about the hidden state, such that dopamine implicitly reflects these beliefs. Here we test this hypothesis in the context of an instrumental task (a two-armed bandit), where the hidden state switches repeatedly. We measured choice behavior and recorded dLight signals reflecting dopamine release in the nucleus accumbens core. Model comparison based on the behavioral data favored models that used Bayesian updating of probabilistic beliefs. These same models also quantitatively matched the dopamine measurements better than non-Bayesian alternatives. We conclude that probabilistic belief computation plays a fundamental role in instrumental performance and associated mesolimbic dopamine signaling.
Collapse
Affiliation(s)
- Albert J. Qü
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
- Center for Computational Biology, University of California, Berkeley, CA, 94720, USA
| | - Lung-Hao Tai
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA
| | - Christopher D. Hall
- Sainsbury Wellcome Centre for Neural Circuits and Behaviour, University College London, London, W1T 4JG, UK
| | - Emilie M. Tu
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
| | | | - Karyna Mishchanchuk
- Department of Neuroscience, Physiology and Pharmacology, University College London, UK
| | - Wan Chen Lin
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA
| | - Juliana B. Chase
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
| | - Andrew F. MacAskill
- Department of Neuroscience, Physiology and Pharmacology, University College London, UK
| | - Anne G.E. Collins
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA
| | - Samuel J. Gershman
- Department of Psychology and Center for Brain Science, Harvard University, Cambridge, MA, 02138, USA
- Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Linda Wilbrecht
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA
| |
Collapse
|
13
|
Luna R, Vadillo MA, Luque D. Model-free decision making resists improved instructions and is enhanced by stimulus-response associations. Cortex 2023; 168:102-113. [PMID: 37690266 DOI: 10.1016/j.cortex.2023.06.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Revised: 05/16/2023] [Accepted: 06/20/2023] [Indexed: 09/12/2023]
Abstract
Human behaviour may be thought of as supported by two different computational-learning mechanisms, model-free and model-based respectively. In model-free strategies, stimulus-response associations are strengthened when actions are followed by a reward and weakened otherwise. In model-based learning, previous to selecting an action, the current values of the different possible actions are computed based on a detailed model of the environment. Previous research with the two-stage task suggests that participants' behaviour usually shows a mixture of both strategies. But, interestingly, a recent study by da Silva and Hare (2020) found that participants primarily deploy model-based behaviour when they are given detailed instructions about the structure of the task. In the present study, we reproduce this essential experiment. Our results confirm that improved instructions give rise to a stronger model-based component. Crucially, we also found a significant effect of reward that became stronger under conditions that favoured the development of strong stimulus-response associations. This suggests that the effect of reward, often taken as indicator of a model-free component, is related to stimulus-response learning.
Collapse
Affiliation(s)
- Raúl Luna
- Institute of Optics, Spanish National Research Council (CSIC), Spain.
| | - Miguel A Vadillo
- Department of Basic Psychology, Faculty of Psychology, Universidad Autónoma de Madrid, Spain
| | - David Luque
- Department of Basic Psychology and Speech Therapy, Faculty of Psychology, Universidad de Málaga, Spain.
| |
Collapse
|
14
|
Schwartenbeck P, Baram A, Liu Y, Mark S, Muller T, Dolan R, Botvinick M, Kurth-Nelson Z, Behrens T. Generative replay underlies compositional inference in the hippocampal-prefrontal circuit. Cell 2023; 186:4885-4897.e14. [PMID: 37804832 PMCID: PMC10914680 DOI: 10.1016/j.cell.2023.09.004] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Revised: 01/23/2023] [Accepted: 09/06/2023] [Indexed: 10/09/2023]
Abstract
Human reasoning depends on reusing pieces of information by putting them together in new ways. However, very little is known about how compositional computation is implemented in the brain. Here, we ask participants to solve a series of problems that each require constructing a whole from a set of elements. With fMRI, we find that representations of novel constructed objects in the frontal cortex and hippocampus are relational and compositional. With MEG, we find that replay assembles elements into compounds, with each replay sequence constituting a hypothesis about a possible configuration of elements. The content of sequences evolves as participants solve each puzzle, progressing from predictable to uncertain elements and gradually converging on the correct configuration. Together, these results suggest a computational bridge between apparently distinct functions of hippocampal-prefrontal circuitry and a role for generative replay in compositional inference and hypothesis testing.
Collapse
Affiliation(s)
- Philipp Schwartenbeck
- University of Tübingen, Tübingen, Germany; Max Planck Institute for Biological Cybernetics, Tübingen, Baden-Württemberg, Germany; Wellcome Trust Centre for Neuroimaging, University College London, London WC1N 3AR, UK; Wellcome Centre for Integrative Neuroimaging, University of Oxford, John Radcliffe Hospital, Oxford OX3 9DU, UK.
| | - Alon Baram
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, John Radcliffe Hospital, Oxford OX3 9DU, UK
| | - Yunzhe Liu
- State Key Laboratory of Cognitive Neuroscience and Learning, IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China; Chinese Institute for Brain Research, Beijing, China
| | - Shirley Mark
- Wellcome Trust Centre for Neuroimaging, University College London, London WC1N 3AR, UK
| | - Timothy Muller
- Wellcome Centre for Integrative Neuroimaging, University of Oxford, John Radcliffe Hospital, Oxford OX3 9DU, UK; Institute of Neurology, University College London, London WC1N 3BG, UK
| | - Raymond Dolan
- Wellcome Trust Centre for Neuroimaging, University College London, London WC1N 3AR, UK; State Key Laboratory of Cognitive Neuroscience and Learning, IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China; Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, London, UK; Department of Psychiatry, Universitätsmedizin Berlin (Campus Charité Mitte), Berlin, Germany
| | - Matthew Botvinick
- Google DeepMind, London, UK; Gatsby Computational Neuroscience Unit, University College London, London, UK
| | - Zeb Kurth-Nelson
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, London, UK; Google DeepMind, London, UK
| | - Timothy Behrens
- Wellcome Trust Centre for Neuroimaging, University College London, London WC1N 3AR, UK; Wellcome Centre for Integrative Neuroimaging, University of Oxford, John Radcliffe Hospital, Oxford OX3 9DU, UK; Sainsbury Wellcome Centre for Neural Circuits and Behaviour, UCL, London W1T 4JG, UK
| |
Collapse
|
15
|
Kim SJ, Affan RO, Frostig H, Scott BB, Alexander AS. Advances in cellular resolution microscopy for brain imaging in rats. NEUROPHOTONICS 2023; 10:044304. [PMID: 38076724 PMCID: PMC10704261 DOI: 10.1117/1.nph.10.4.044304] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 09/23/2023] [Accepted: 11/07/2023] [Indexed: 02/12/2024]
Abstract
Rats are used in neuroscience research because of their physiological similarities with humans and accessibility as model organisms, trainability, and behavioral repertoire. In particular, rats perform a wide range of sophisticated social, cognitive, motor, and learning behaviors within the contexts of both naturalistic and laboratory environments. Further progress in neuroscience can be facilitated by using advanced imaging methods to measure the complex neural and physiological processes during behavior in rats. However, compared with the mouse, the rat nervous system offers a set of challenges, such as larger brain size, decreased neuron density, and difficulty with head restraint. Here, we review recent advances in in vivo imaging techniques in rats with a special focus on open-source solutions for calcium imaging. Finally, we provide suggestions for both users and developers of in vivo imaging systems for rats.
Collapse
Affiliation(s)
- Su Jin Kim
- Johns Hopkins University, Department of Psychological and Brain Sciences, Baltimore, Maryland, United States
| | - Rifqi O. Affan
- Boston University, Center for Systems Neuroscience, Department of Psychological and Brain Sciences, Boston, Massachusetts, United States
- Boston University, Graduate Program in Neuroscience, Boston, Massachusetts, United States
| | - Hadas Frostig
- Boston University, Center for Systems Neuroscience, Department of Psychological and Brain Sciences, Boston, Massachusetts, United States
| | - Benjamin B. Scott
- Boston University, Center for Systems Neuroscience, Department of Psychological and Brain Sciences, Boston, Massachusetts, United States
- Boston University, Neurophotonics Center and Photonics Center, Boston, Massachusetts, United States
| | - Andrew S. Alexander
- University of California Santa Barbara, Department of Psychological and Brain Sciences, Santa Barbara, California, United States
| |
Collapse
|
16
|
Mehrotra D, Dubé L. Accounting for multiscale processing in adaptive real-world decision-making via the hippocampus. Front Neurosci 2023; 17:1200842. [PMID: 37732307 PMCID: PMC10508350 DOI: 10.3389/fnins.2023.1200842] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 08/25/2023] [Indexed: 09/22/2023] Open
Abstract
For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure. The emphasis of examining the neural substrates of value-based decision making has been on the striatum and prefrontal regions, especially with regards to the "here and now" decision-making. Yet, such a dichotomy does not embrace all the dynamic complexity involved. In addition, despite robust research on the role of the hippocampus in memory and spatial learning, its contribution to value-based decision making is just starting to be explored. This paper aims to better appreciate the role of the hippocampus in decision-making and advance the successor representation (SR) as a candidate mechanism for encoding state representations in the hippocampus, separate from reward representations. To this end, we review research that relates hippocampal sequences to SR models showing that the implementation of such sequences in reinforcement learning agents improves their performance. This also enables the agents to perform multiscale temporal processing in a biologically plausible manner. Altogether, we articulate a framework to advance current striatal and prefrontal-focused decision making to better account for multiscale mechanisms underlying various real-world time-related concepts such as the self that cumulates over a person's life course.
Collapse
Affiliation(s)
- Dhruv Mehrotra
- Integrated Program in Neuroscience, McGill University, Montréal, QC, Canada
- Montréal Neurological Institute, McGill University, Montréal, QC, Canada
| | - Laurette Dubé
- Desautels Faculty of Management, McGill University, Montréal, QC, Canada
- McGill Center for the Convergence of Health and Economics, McGill University, Montréal, QC, Canada
| |
Collapse
|
17
|
Le NM, Yildirim M, Wang Y, Sugihara H, Jazayeri M, Sur M. Mixtures of strategies underlie rodent behavior during reversal learning. PLoS Comput Biol 2023; 19:e1011430. [PMID: 37708113 PMCID: PMC10501641 DOI: 10.1371/journal.pcbi.1011430] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2023] [Accepted: 08/09/2023] [Indexed: 09/16/2023] Open
Abstract
In reversal learning tasks, the behavior of humans and animals is often assumed to be uniform within single experimental sessions to facilitate data analysis and model fitting. However, behavior of agents can display substantial variability in single experimental sessions, as they execute different blocks of trials with different transition dynamics. Here, we observed that in a deterministic reversal learning task, mice display noisy and sub-optimal choice transitions even at the expert stages of learning. We investigated two sources of the sub-optimality in the behavior. First, we found that mice exhibit a high lapse rate during task execution, as they reverted to unrewarded directions after choice transitions. Second, we unexpectedly found that a majority of mice did not execute a uniform strategy, but rather mixed between several behavioral modes with different transition dynamics. We quantified the use of such mixtures with a state-space model, block Hidden Markov Model (block HMM), to dissociate the mixtures of dynamic choice transitions in individual blocks of trials. Additionally, we found that blockHMM transition modes in rodent behavior can be accounted for by two different types of behavioral algorithms, model-free or inference-based learning, that might be used to solve the task. Combining these approaches, we found that mice used a mixture of both exploratory, model-free strategies and deterministic, inference-based behavior in the task, explaining their overall noisy choice sequences. Together, our combined computational approach highlights intrinsic sources of noise in rodent reversal learning behavior and provides a richer description of behavior than conventional techniques, while uncovering the hidden states that underlie the block-by-block transitions.
Collapse
Affiliation(s)
- Nhat Minh Le
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
| | - Murat Yildirim
- Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Department of Neurosciences, Cleveland Clinic Lerner Research Institute, Cleveland, Ohio, United States of America
| | - Yizhi Wang
- Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
| | - Hiroki Sugihara
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
| | - Mehrdad Jazayeri
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
| | - Mriganka Sur
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
- Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, Massachusetts, United States of America
| |
Collapse
|
18
|
Jankowski MM, Polterovich A, Kazakov A, Niediek J, Nelken I. An automated, low-latency environment for studying the neural basis of behavior in freely moving rats. BMC Biol 2023; 21:172. [PMID: 37568111 PMCID: PMC10416379 DOI: 10.1186/s12915-023-01660-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Accepted: 07/10/2023] [Indexed: 08/13/2023] Open
Abstract
BACKGROUND Behavior consists of the interaction between an organism and its environment, and is controlled by the brain. Brain activity varies at sub-second time scales, but behavioral measures are usually coarse (often consisting of only binary trial outcomes). RESULTS To overcome this mismatch, we developed the Rat Interactive Foraging Facility (RIFF): a programmable interactive arena for freely moving rats with multiple feeding areas, multiple sound sources, high-resolution behavioral tracking, and simultaneous electrophysiological recordings. The paper provides detailed information about the construction of the RIFF and the software used to control it. To illustrate the flexibility of the RIFF, we describe two complex tasks implemented in the RIFF, a foraging task and a sound localization task. Rats quickly learned to obtain rewards in both tasks. Neurons in the auditory cortex as well as neurons in the auditory field in the posterior insula had sound-driven activity during behavior. Remarkably, neurons in both structures also showed sensitivity to non-auditory parameters such as location in the arena and head-to-body angle. CONCLUSIONS The RIFF provides insights into the cognitive capabilities and learning mechanisms of rats and opens the way to a better understanding of how brains control behavior. The ability to do so depends crucially on the combination of wireless electrophysiology and detailed behavioral documentation available in the RIFF.
Collapse
Affiliation(s)
- Maciej M Jankowski
- The Edmond and Lily Safra Center for Brain Sciences and the Department of Neurobiology, Silberman Institute of Life Sciences, the Hebrew University of Jerusalem, Jerusalem, Israel
- BioTechMed Center, Multimedia Systems Department, Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gdansk, Poland
| | - Ana Polterovich
- The Edmond and Lily Safra Center for Brain Sciences and the Department of Neurobiology, Silberman Institute of Life Sciences, the Hebrew University of Jerusalem, Jerusalem, Israel
| | - Alex Kazakov
- The Edmond and Lily Safra Center for Brain Sciences and the Department of Neurobiology, Silberman Institute of Life Sciences, the Hebrew University of Jerusalem, Jerusalem, Israel
| | - Johannes Niediek
- The Edmond and Lily Safra Center for Brain Sciences and the Department of Neurobiology, Silberman Institute of Life Sciences, the Hebrew University of Jerusalem, Jerusalem, Israel
| | - Israel Nelken
- The Edmond and Lily Safra Center for Brain Sciences and the Department of Neurobiology, Silberman Institute of Life Sciences, the Hebrew University of Jerusalem, Jerusalem, Israel.
| |
Collapse
|
19
|
De Beukelaer S, Sokolov AA, Müri RM. Case report: "Proust phenomenon" after right posterior cerebral artery occlusion. Front Neurol 2023; 14:1183265. [PMID: 37521297 PMCID: PMC10374343 DOI: 10.3389/fneur.2023.1183265] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Accepted: 06/19/2023] [Indexed: 08/01/2023] Open
Abstract
Odors evoking vivid and intensely felt autobiographical memories are known as the "Proust phenomenon," delineating the particularity of olfaction in being more effective with eliciting emotional memories than other sensory modalities. The phenomenon has been described extensively in healthy participants as well as in patients during pre-epilepsy surgery evaluation after focal stimulation of the amygdalae and post-traumatic stress disorder (PTSD). In this study, we provide the inaugural description of aversive odor-evoked autobiographical memories after stroke in the right hippocampal, parahippocampal, and thalamic nuclei. As potential underlying neural signatures of the phenomenon, we discuss the disinhibition of limbic circuits and impaired communication between the major networks, such as saliency, central executive, and default mode network.
Collapse
Affiliation(s)
- Sophie De Beukelaer
- Department of Neurology, University Hospital, Inselspital Bern, Bern, Switzerland
| | - A. A. Sokolov
- Service de Neuropsychologie et de Neuroréhabilitation, Département des Neurosciences Cliniques, Centre Hospitalier Universitaire Vaudois (CHUV), Lausanne, Switzerland
| | - R. M. Müri
- Department of Neurology, University Hospital, Inselspital Bern, Bern, Switzerland
- Gerontechnology and Rehabilitation Group, ARTORG Center, University of Bern, Bern, Switzerland
| |
Collapse
|
20
|
van Opheusden B, Kuperwajs I, Galbiati G, Bnaya Z, Li Y, Ma WJ. Expertise increases planning depth in human gameplay. Nature 2023; 618:1000-1005. [PMID: 37258667 DOI: 10.1038/s41586-023-06124-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2021] [Accepted: 04/24/2023] [Indexed: 06/02/2023]
Abstract
A hallmark of human intelligence is the ability to plan multiple steps into the future1,2. Despite decades of research3-5, it is still debated whether skilled decision-makers plan more steps ahead than novices6-8. Traditionally, the study of expertise in planning has used board games such as chess, but the complexity of these games poses a barrier to quantitative estimates of planning depth. Conversely, common planning tasks in cognitive science often have a lower complexity9,10 and impose a ceiling for the depth to which any player can plan. Here we investigate expertise in a complex board game that offers ample opportunity for skilled players to plan deeply. We use model fitting methods to show that human behaviour can be captured using a computational cognitive model based on heuristic search. To validate this model, we predict human choices, response times and eye movements. We also perform a Turing test and a reconstruction experiment. Using the model, we find robust evidence for increased planning depth with expertise in both laboratory and large-scale mobile data. Experts memorize and reconstruct board features more accurately. Using complex tasks combined with precise behavioural modelling might expand our understanding of human planning and help to bridge the gap with progress in artificial intelligence.
Collapse
Affiliation(s)
- Bas van Opheusden
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA.
- Department of Computer Science, Princeton University, Princeton, NJ, USA.
| | - Ionatan Kuperwajs
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA
| | - Gianni Galbiati
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA
- Vidrovr, New York, NY, USA
| | - Zahy Bnaya
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA
| | - Yunqi Li
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA
| | - Wei Ji Ma
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA
| |
Collapse
|
21
|
Feher da Silva C, Lombardi G, Edelson M, Hare TA. Rethinking model-based and model-free influences on mental effort and striatal prediction errors. Nat Hum Behav 2023:10.1038/s41562-023-01573-1. [PMID: 37012365 DOI: 10.1038/s41562-023-01573-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Accepted: 02/27/2023] [Indexed: 04/05/2023]
Abstract
A standard assumption in neuroscience is that low-effort model-free learning is automatic and continuously used, whereas more complex model-based strategies are only used when the rewards they generate are worth the additional effort. We present evidence refuting this assumption. First, we demonstrate flaws in previous reports of combined model-free and model-based reward prediction errors in the ventral striatum that probably led to spurious results. More appropriate analyses yield no evidence of model-free prediction errors in this region. Second, we find that task instructions generating more correct model-based behaviour reduce rather than increase mental effort. This is inconsistent with cost-benefit arbitration between model-based and model-free strategies. Together, our data indicate that model-free learning may not be automatic. Instead, humans can reduce mental effort by using a model-based strategy alone rather than arbitrating between multiple strategies. Our results call for re-evaluation of the assumptions in influential theories of learning and decision-making.
Collapse
Affiliation(s)
| | - Gaia Lombardi
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
| | - Micah Edelson
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
| | - Todd A Hare
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland.
| |
Collapse
|
22
|
Hales CA, Clark L, Winstanley CA. Computational approaches to modeling gambling behaviour: Opportunities for understanding disordered gambling. Neurosci Biobehav Rev 2023; 147:105083. [PMID: 36758827 DOI: 10.1016/j.neubiorev.2023.105083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 01/05/2023] [Accepted: 02/06/2023] [Indexed: 02/10/2023]
Abstract
Computational modeling has become an important tool in neuroscience and psychiatry research to provide insight into the cognitive processes underlying normal and pathological behavior. There are two modeling frameworks, reinforcement learning (RL) and drift diffusion modeling (DDM), that are well-developed in cognitive science, and have begun to be applied to Gambling Disorder. RL models focus on explaining how an agent uses reward to learn about the environment and make decisions based on outcomes. The DDM is a binary choice framework that breaks down decision making into psychologically meaningful components based on choice reaction time analyses. Both approaches have begun to yield insight into aspects of cognition that are important for, but not unique to, gambling, and thus relevant to the development of Gambling Disorder. However, these approaches also oversimplify or neglect various aspects of decision making seen in real-world gambling behavior. Gambling Disorder presents an opportunity for 'bespoke' modeling approaches to consider these neglected components. In this review, we discuss studies that have used RL and DDM frameworks to investigate some of the key cognitive components in gambling and Gambling Disorder. We also include an overview of Bayesian models, a methodology that could be useful for more tailored modeling approaches. We highlight areas in which computational modeling could enable progression in the investigation of the cognitive mechanisms relevant to gambling.
Collapse
Affiliation(s)
- C A Hales
- Djavad Mowafaghian Centre for Brain Health, University of British Columbia, Vancouver, British Columbia, Canada; Department of Psychology, University of British Columbia, Vancouver, British Columbia, Canada.
| | - L Clark
- Djavad Mowafaghian Centre for Brain Health, University of British Columbia, Vancouver, British Columbia, Canada; Department of Psychology, University of British Columbia, Vancouver, British Columbia, Canada
| | - C A Winstanley
- Djavad Mowafaghian Centre for Brain Health, University of British Columbia, Vancouver, British Columbia, Canada; Department of Psychology, University of British Columbia, Vancouver, British Columbia, Canada
| |
Collapse
|
23
|
Brandl F, Knolle F, Avram M, Leucht C, Yakushev I, Priller J, Leucht S, Ziegler S, Wunderlich K, Sorg C. Negative symptoms, striatal dopamine and model-free reward decision-making in schizophrenia. Brain 2023; 146:767-777. [PMID: 35875972 DOI: 10.1093/brain/awac268] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2022] [Revised: 06/13/2022] [Accepted: 07/04/2022] [Indexed: 11/13/2022] Open
Abstract
Negative symptoms, such as lack of motivation or social withdrawal, are highly prevalent and debilitating in patients with schizophrenia. Underlying mechanisms of negative symptoms are incompletely understood, thereby preventing the development of targeted treatments. We hypothesized that in patients with schizophrenia during psychotic remission, impaired influences of both model-based and model-free reward predictions on decision-making ('reward prediction influence', RPI) underlie negative symptoms. We focused on psychotic remission, because psychotic symptoms might confound reward-based decision-making. Moreover, we hypothesized that impaired model-based/model-free RPIs depend on alterations of both associative striatum dopamine synthesis and storage (DSS) and executive functioning. Both factors influence RPI in healthy subjects and are typically impaired in schizophrenia. Twenty-five patients with schizophrenia with pronounced negative symptoms during psychotic remission and 24 healthy controls were included in the study. Negative symptom severity was measured by the Positive and Negative Syndrome Scale negative subscale, model-based/model-free RPI by the two-stage decision task, associative striatum DSS by 18F-DOPA positron emission tomography and executive functioning by the symbol coding task. Model-free RPI was selectively reduced in patients and associated with negative symptom severity as well as with reduced associative striatum DSS (in patients only) and executive functions (both in patients and controls). In contrast, model-based RPI was not altered in patients. Results provide evidence for impaired model-free reward prediction influence as a mechanism for negative symptoms in schizophrenia as well as for reduced associative striatum dopamine and executive dysfunction as relevant factors. Data suggest potential treatment targets for patients with schizophrenia and pronounced negative symptoms.
Collapse
Affiliation(s)
- Felix Brandl
- Department of Psychiatry and Psychotherapy, School of Medicine, Technical University of Munich, Munich, 81675, Germany.,Department of Neuroradiology, School of Medicine, Technical University of Munich, Munich, 81675, Germany.,TUM-NIC Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, 81675, Germany
| | - Franziska Knolle
- Department of Neuroradiology, School of Medicine, Technical University of Munich, Munich, 81675, Germany.,TUM-NIC Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, 81675, Germany.,Department of Psychiatry, University of Cambridge, Cambridge CB20SZ, UK
| | - Mihai Avram
- Translational Psychiatry, Department of Psychiatry and Psychotherapy, University of Lübeck, Lübeck, 23538, Germany
| | - Claudia Leucht
- Department of Psychiatry and Psychotherapy, School of Medicine, Technical University of Munich, Munich, 81675, Germany
| | - Igor Yakushev
- Department of Nuclear Medicine, School of Medicine, Technical University of Munich, Munich, 81675, Germany
| | - Josef Priller
- Department of Psychiatry and Psychotherapy, School of Medicine, Technical University of Munich, Munich, 81675, Germany.,Neuropsychiatry, Charité-Universitätsmedizin Berlin, and DZNE, Berlin, 10117, Germany.,UK DRI at University of Edinburgh, Edinburgh EH16 4SB, UK.,IoPPN, King's College London, London SE5 8AF, UK
| | - Stefan Leucht
- Department of Psychiatry and Psychotherapy, School of Medicine, Technical University of Munich, Munich, 81675, Germany.,Department of Psychosis studies, King's College London, London, UK
| | - Sibylle Ziegler
- Department of Nuclear Medicine, Ludwig-Maximilians University Munich, Munich, 81377, Germany
| | - Klaus Wunderlich
- Department of Psychology, Ludwig-Maximilians University Munich, Munich, 81377, Germany
| | - Christian Sorg
- Department of Psychiatry and Psychotherapy, School of Medicine, Technical University of Munich, Munich, 81675, Germany.,Department of Neuroradiology, School of Medicine, Technical University of Munich, Munich, 81675, Germany.,TUM-NIC Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, 81675, Germany
| |
Collapse
|
24
|
Wimmer GE, Liu Y, McNamee DC, Dolan RJ. Distinct replay signatures for prospective decision-making and memory preservation. Proc Natl Acad Sci U S A 2023; 120:e2205211120. [PMID: 36719914 PMCID: PMC9963918 DOI: 10.1073/pnas.2205211120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Accepted: 12/05/2022] [Indexed: 02/01/2023] Open
Abstract
Theories of neural replay propose that it supports a range of functions, most prominently planning and memory consolidation. Here, we test the hypothesis that distinct signatures of replay in the same task are related to model-based decision-making ("planning") and memory preservation. We designed a reward learning task wherein participants utilized structure knowledge for model-based evaluation, while at the same time had to maintain knowledge of two independent and randomly alternating task environments. Using magnetoencephalography and multivariate analysis, we first identified temporally compressed sequential reactivation, or replay, both prior to choice and following reward feedback. Before choice, prospective replay strength was enhanced for the current task-relevant environment when a model-based planning strategy was beneficial. Following reward receipt, and consistent with a memory preservation role, replay for the alternative distal task environment was enhanced as a function of decreasing recency of experience with that environment. Critically, these planning and memory preservation relationships were selective to pre-choice and post-feedback periods, respectively. Our results provide support for key theoretical proposals regarding the functional role of replay and demonstrate that the relative strength of planning and memory-related signals are modulated by ongoing computational and task demands.
Collapse
Affiliation(s)
- G. Elliott Wimmer
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, LondonWC1B 5EH, UK
- Wellcome Centre for Human Neuroimaging, University College London, LondonWC1N 3BG, UK
| | - Yunzhe Liu
- State Key Laboratory of Cognitive Neuroscience and Learning, IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing100875, China
- Chinese Institute for Brain Research, Beijing100875, China
| | - Daniel C. McNamee
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, LondonWC1B 5EH, UK
- Wellcome Centre for Human Neuroimaging, University College London, LondonWC1N 3BG, UK
- Neuroscience Programme, Champalimaud Research, Lisbon1400-038, Portugal
| | - Raymond J. Dolan
- Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, LondonWC1B 5EH, UK
- Wellcome Centre for Human Neuroimaging, University College London, LondonWC1N 3BG, UK
- State Key Laboratory of Cognitive Neuroscience and Learning, IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing100875, China
| |
Collapse
|
25
|
Oguchi M, Li Y, Matsumoto Y, Kiyonari T, Yamamoto K, Sugiura S, Sakagami M. Proselfs depend more on model-based than model-free learning in a non-social probabilistic state-transition task. Sci Rep 2023; 13:1419. [PMID: 36697448 PMCID: PMC9876908 DOI: 10.1038/s41598-023-27609-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Accepted: 01/04/2023] [Indexed: 01/26/2023] Open
Abstract
Humans form complex societies in which we routinely engage in social decision-making regarding the allocation of resources among ourselves and others. One dimension that characterizes social decision-making in particular is whether to prioritize self-interest or respect for others-proself or prosocial. What causes this individual difference in social value orientation? Recent developments in the social dual-process theory argue that social decision-making is characterized by its underlying domain-general learning systems: the model-free and model-based systems. In line with this "learning" approach, we propose and experimentally test the hypothesis that differences in social preferences stem from which learning system is dominant in an individual. Here, we used a non-social state transition task that allowed us to assess the balance between model-free/model-based learning and investigate its relation to the social value orientations. The results showed that proselfs depended more on model-based learning, whereas prosocials depended more on model-free learning. Reward amount and reaction time analyses showed that proselfs learned the task structure earlier in the session than prosocials, reflecting their difference in model-based/model-free learning dependence. These findings support the learning hypothesis on what makes differences in social preferences and have implications for understanding the mechanisms of prosocial behavior.
Collapse
Affiliation(s)
- Mineki Oguchi
- Brain Science Institute, Tamagawa University, 6-1-1, Tamagawagakuen, Machida, Tokyo, Japan
| | - Yang Li
- Brain Science Institute, Tamagawa University, 6-1-1, Tamagawagakuen, Machida, Tokyo, Japan.,Graduate School of Informatics, Nagoya University, Nagoya, Japan
| | - Yoshie Matsumoto
- Brain Science Institute, Tamagawa University, 6-1-1, Tamagawagakuen, Machida, Tokyo, Japan.,Department of Psychology, Faculty of Human Sciences, Seinan Gakuin University, Fukuoka, Japan
| | - Toko Kiyonari
- School of Social Informatics, Aoyama Gakuin University, Kanagawa, Japan
| | | | | | - Masamichi Sakagami
- Brain Science Institute, Tamagawa University, 6-1-1, Tamagawagakuen, Machida, Tokyo, Japan.
| |
Collapse
|
26
|
Moin Afshar N, Cinotti F, Martin D, Khamassi M, Calu DJ, Taylor JR, Groman SM. Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related. J Neurosci 2023; 43:458-471. [PMID: 36216504 PMCID: PMC9864557 DOI: 10.1523/jneurosci.1113-22.2022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Revised: 10/03/2022] [Accepted: 10/06/2022] [Indexed: 01/25/2023] Open
Abstract
Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.
Collapse
Affiliation(s)
- Neema Moin Afshar
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
| | - François Cinotti
- Department of Experimental Psychology, University of Oxford, Oxford OX2 6GG, United Kingdom
| | - David Martin
- Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
| | - Mehdi Khamassi
- Institute of Intelligent Systems and Robotics, Centre National de la Recherche Scientifique, Sorbonne University, 75005 Paris, France
| | - Donna J Calu
- Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
- Program in Neuroscience, University of Maryland School of Medicine, Baltimore, Maryland 21201
| | - Jane R Taylor
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
- Department of Psychology, Yale University, New Haven, Connecticut 06520
| | - Stephanie M Groman
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
- Department of Neuroscience, University of Minnesota Medical School, Minneapolis, Minnesota 55455
- Department of Psychology, University of Minnesota, Minneapolis, Minnesota 55455
| |
Collapse
|
27
|
Makino H. Arithmetic value representation for hierarchical behavior composition. Nat Neurosci 2023; 26:140-149. [PMID: 36550292 PMCID: PMC9829535 DOI: 10.1038/s41593-022-01211-5] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2021] [Accepted: 10/21/2022] [Indexed: 12/24/2022]
Abstract
The ability to compose new skills from a preacquired behavior repertoire is a hallmark of biological intelligence. Although artificial agents extract reusable skills from past experience and recombine them in a hierarchical manner, whether the brain similarly composes a novel behavior is largely unknown. In the present study, I show that deep reinforcement learning agents learn to solve a novel composite task by additively combining representations of prelearned action values of constituent subtasks. Learning efficacy in the composite task was further augmented by the introduction of stochasticity in behavior during pretraining. These theoretical predictions were empirically tested in mice, where subtask pretraining enhanced learning of the composite task. Cortex-wide, two-photon calcium imaging revealed analogous neural representations of combined action values, with improved learning when the behavior variability was amplified. Together, these results suggest that the brain composes a novel behavior with a simple arithmetic operation of preacquired action-value representations with stochastic policies.
Collapse
Affiliation(s)
- Hiroshi Makino
- Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore.
| |
Collapse
|
28
|
Comrie AE, Frank LM, Kay K. Imagination as a fundamental function of the hippocampus. Philos Trans R Soc Lond B Biol Sci 2022; 377:20210336. [PMID: 36314152 PMCID: PMC9620759 DOI: 10.1098/rstb.2021.0336] [Citation(s) in RCA: 18] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Accepted: 04/20/2022] [Indexed: 08/25/2023] Open
Abstract
Imagination is a biological function that is vital to human experience and advanced cognition. Despite this importance, it remains unknown how imagination is realized in the brain. Substantial research focusing on the hippocampus, a brain structure traditionally linked to memory, indicates that firing patterns in spatially tuned neurons can represent previous and upcoming paths in space. This work has generally been interpreted under standard views that the hippocampus implements cognitive abilities primarily related to actual experience, whether in the past (e.g. recollection, consolidation), present (e.g. spatial mapping) or future (e.g. planning). However, relatively recent findings in rodents identify robust patterns of hippocampal firing corresponding to a variety of alternatives to actual experience, in many cases without overt reference to the past, present or future. Given these findings, and others on hippocampal contributions to human imagination, we suggest that a fundamental function of the hippocampus is to generate a wealth of hypothetical experiences and thoughts. Under this view, traditional accounts of hippocampal function in episodic memory and spatial navigation can be understood as particular applications of a more general system for imagination. This view also suggests that the hippocampus contributes to a wider range of cognitive abilities than previously thought. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.
Collapse
Affiliation(s)
- Alison E. Comrie
- Neuroscience Graduate Program, University of California San Francisco, 675 Nelson Rising Lane, San Francisco, CA 94158, USA
- Kavli Institute for Fundamental Neuroscience, University of California San Francisco, 675 Nelson Rising Lane, San Francisco, CA 94158, USA
- Center for Integrative Neuroscience, University of California San Francisco, 675 Nelson Rising Lane, San Francisco, CA 94158, USA
- Departments of Physiology and Psychiatry, University of California San Francisco, 675 Nelson Rising Lane, San Francisco, CA 94158, USA
| | - Loren M. Frank
- Kavli Institute for Fundamental Neuroscience, University of California San Francisco, 675 Nelson Rising Lane, San Francisco, CA 94158, USA
- Center for Integrative Neuroscience, University of California San Francisco, 675 Nelson Rising Lane, San Francisco, CA 94158, USA
- Departments of Physiology and Psychiatry, University of California San Francisco, 675 Nelson Rising Lane, San Francisco, CA 94158, USA
- Howard Hughes Medical Institute, University of California San Francisco, 675 Nelson Rising Lane, San Francisco, CA 94158, USA
| | - Kenneth Kay
- Zuckerman Institute, Center for Theoretical Neuroscience, Columbia University, 3227 Broadway, New York, NY 10027, USA
| |
Collapse
|
29
|
Prospective and retrospective values integrated in frontal cortex drive predictive choice. Proc Natl Acad Sci U S A 2022; 119:e2206067119. [PMID: 36417435 PMCID: PMC9889848 DOI: 10.1073/pnas.2206067119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
To make a deliberate action in a volatile environment, the brain must frequently reassess the value of each action (action-value). Choice can be initially made from the experience of trial-and-errors, but once the dynamics of the environment is learned, the choice can be made from the knowledge of the environment. The action-values constructed from the experience (retrospective value) and the ones from the knowledge (prospective value) were identified in various regions of the brain. However, how and which neural circuit integrates these values and executes the chosen action remains unknown. Combining reinforcement learning and two-photon calcium imaging, we found that the preparatory activity of neurons in a part of the frontal cortex, the anterior-lateral motor (ALM) area, initially encodes retrospective value, but after extensive training, they jointly encode the retrospective and prospective value. Optogenetic inhibition of ALM preparatory activity specifically abolished the expert mice's predictive choice behavior and returned them to the novice-like state. Thus, the integrated action-value encoded in the preparatory activity of ALM plays an important role to bias the action toward the knowledge-dependent, predictive choice behavior.
Collapse
|
30
|
Pearce AL, Fuchs BA, Keller KL. The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors. Front Nutr 2022; 9:1021868. [PMID: 36483928 PMCID: PMC9722736 DOI: 10.3389/fnut.2022.1021868] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2022] [Accepted: 11/04/2022] [Indexed: 11/23/2022] Open
Abstract
The obesogenic food environment includes easy access to highly-palatable, energy-dense, "ultra-processed" foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption-the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.
Collapse
Affiliation(s)
- Alaina L. Pearce
- Social Science Research Institute, Pennsylvania State University, University Park, PA, United States
- Department of Nutritional Sciences, Pennsylvania State University, University Park, PA, United States
| | - Bari A. Fuchs
- Department of Nutritional Sciences, Pennsylvania State University, University Park, PA, United States
| | - Kathleen L. Keller
- Social Science Research Institute, Pennsylvania State University, University Park, PA, United States
- Department of Nutritional Sciences, Pennsylvania State University, University Park, PA, United States
- Department of Food Science, Pennsylvania State University, University Park, PA, United States
| |
Collapse
|
31
|
Ujfalussy BB, Orbán G. Sampling motion trajectories during hippocampal theta sequences. eLife 2022; 11:74058. [DOI: 10.7554/elife.74058] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2021] [Accepted: 09/28/2022] [Indexed: 11/06/2022] Open
Abstract
Efficient planning in complex environments requires that uncertainty associated with current inferences and possible consequences of forthcoming actions is represented. Representation of uncertainty has been established in sensory systems during simple perceptual decision making tasks but it remains unclear if complex cognitive computations such as planning and navigation are also supported by probabilistic neural representations. Here, we capitalized on gradually changing uncertainty along planned motion trajectories during hippocampal theta sequences to capture signatures of uncertainty representation in population responses. In contrast with prominent theories, we found no evidence of encoding parameters of probability distributions in the momentary population activity recorded in an open-field navigation task in rats. Instead, uncertainty was encoded sequentially by sampling motion trajectories randomly and efficiently in subsequent theta cycles from the distribution of potential trajectories. Our analysis is the first to demonstrate that the hippocampus is well equipped to contribute to optimal planning by representing uncertainty.
Collapse
Affiliation(s)
- Balazs B Ujfalussy
- Laboratory of Biological Computation, Institute of Experimental Medicine
- Laboratory of Neuronal Signalling, Institute of Experimental Medicine, Budapest
| | - Gergő Orbán
- Computational Systems Neuroscience Lab, Wigner Research Center for Physics, Budapest
| |
Collapse
|
32
|
Wassum KM. Amygdala-cortical collaboration in reward learning and decision making. eLife 2022; 11:80926. [PMID: 36062909 PMCID: PMC9444241 DOI: 10.7554/elife.80926] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Accepted: 08/22/2022] [Indexed: 12/16/2022] Open
Abstract
Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.
Collapse
Affiliation(s)
- Kate M Wassum
- Department of Psychology, University of California, Los Angeles, Los Angeles, United States.,Brain Research Institute, University of California, Los Angeles, Los Angeles, United States.,Integrative Center for Learning and Memory, University of California, Los Angeles, Los Angeles, United States.,Integrative Center for Addictive Disorders, University of California, Los Angeles, Los Angeles, United States
| |
Collapse
|
33
|
Miller KJ, Botvinick MM, Brody CD. Value representations in the rodent orbitofrontal cortex drive learning, not choice. eLife 2022; 11:64575. [PMID: 35975792 PMCID: PMC9462853 DOI: 10.7554/elife.64575] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2020] [Accepted: 08/01/2022] [Indexed: 11/13/2022] Open
Abstract
Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice: the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning: expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here, we employ a recently developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.
Collapse
Affiliation(s)
| | | | - Carlos D Brody
- Princeton Neuroscience Institute, Princeton University, Princeton, United States
| |
Collapse
|
34
|
Stoianov I, Maisto D, Pezzulo G. The hippocampal formation as a hierarchical generative model supporting generative replay and continual learning. Prog Neurobiol 2022; 217:102329. [PMID: 35870678 DOI: 10.1016/j.pneurobio.2022.102329] [Citation(s) in RCA: 15] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2021] [Revised: 07/15/2022] [Accepted: 07/19/2022] [Indexed: 11/28/2022]
Abstract
We advance a novel computational theory of the hippocampal formation as a hierarchical generative model that organizes sequential experiences, such as rodent trajectories during spatial navigation, into coherent spatiotemporal contexts. We propose that the hippocampal generative model is endowed with inductive biases to identify individual items of experience (first hierarchical layer), organize them into sequences (second layer) and cluster them into maps (third layer). This theory entails a novel characterization of hippocampal reactivations as generative replay: the offline resampling of fictive sequences from the generative model, which supports the continual learning of multiple sequential experiences. We show that the model learns and efficiently retains multiple spatial navigation trajectories, by organizing them into spatial maps. Furthermore, the model reproduces flexible and prospective aspects of hippocampal dynamics that are challenging to explain within existing frameworks. This theory reconciles multiple roles of the hippocampal formation in map-based navigation, episodic memory and imagination.
Collapse
Affiliation(s)
- Ivilin Stoianov
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy
| | - Domenico Maisto
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy
| | - Giovanni Pezzulo
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy.
| |
Collapse
|
35
|
de Cothi W, Nyberg N, Griesbauer EM, Ghanamé C, Zisch F, Lefort JM, Fletcher L, Newton C, Renaudineau S, Bendor D, Grieves R, Duvelle É, Barry C, Spiers HJ. Predictive maps in rats and humans for spatial navigation. Curr Biol 2022; 32:3676-3689.e5. [PMID: 35863351 DOI: 10.1016/j.cub.2022.06.090] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2022] [Revised: 05/19/2022] [Accepted: 06/29/2022] [Indexed: 11/25/2022]
Abstract
Much of our understanding of navigation comes from the study of individual species, often with specific tasks tailored to those species. Here, we provide a novel experimental and analytic framework integrating across humans, rats, and simulated reinforcement learning (RL) agents to interrogate the dynamics of behavior during spatial navigation. We developed a novel open-field navigation task ("Tartarus maze") requiring dynamic adaptation (shortcuts and detours) to frequently changing obstructions on the path to a hidden goal. Humans and rats were remarkably similar in their trajectories. Both species showed the greatest similarity to RL agents utilizing a "successor representation," which creates a predictive map. Humans also displayed trajectory features similar to model-based RL agents, which implemented an optimal tree-search planning procedure. Our results help refine models seeking to explain mammalian navigation in dynamic environments and highlight the utility of modeling the behavior of different species to uncover the shared mechanisms that support behavior.
Collapse
Affiliation(s)
- William de Cothi
- Department of Cell and Developmental Biology, University College London, London, UK; Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK.
| | - Nils Nyberg
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK
| | - Eva-Maria Griesbauer
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK
| | - Carole Ghanamé
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK
| | - Fiona Zisch
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK; The Bartlett School of Architecture, University College London, London, UK
| | - Julie M Lefort
- Department of Cell and Developmental Biology, University College London, London, UK
| | - Lydia Fletcher
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK
| | - Coco Newton
- Department of Clinical Neurosciences, University of Cambridge, Cambridge, UK
| | - Sophie Renaudineau
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK
| | - Daniel Bendor
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK
| | - Roddy Grieves
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK; Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Éléonore Duvelle
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK; Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH, USA
| | - Caswell Barry
- Department of Cell and Developmental Biology, University College London, London, UK
| | - Hugo J Spiers
- Institute of Behavioral Neuroscience, Department of Experimental Psychology, Division of Psychology and Language Sciences, University College London, London, UK.
| |
Collapse
|
36
|
Glitz L, Juechems K, Summerfield C, Garrett N. Model Sharing in the Human Medial Temporal Lobe. J Neurosci 2022; 42:5410-5426. [PMID: 35606146 PMCID: PMC7613027 DOI: 10.1523/jneurosci.1978-21.2022] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2021] [Revised: 04/20/2022] [Accepted: 04/23/2022] [Indexed: 11/21/2022] Open
Abstract
Effective planning involves knowing where different actions take us. However, natural environments are rich and complex, leading to an exponential increase in memory demand as a plan grows in depth. One potential solution is to filter out features of the environment irrelevant to the task at hand. This enables a shared model of transition dynamics to be used for planning over a range of different input features. Here, we asked human participants (13 male, 16 female) to perform a sequential decision-making task, designed so that knowledge should be integrated independently of the input features (visual cues) present in one case but not in another. Participants efficiently switched between using a low-dimensional (cue independent) and a high-dimensional (cue specific) representation of state transitions. fMRI data identified the medial temporal lobe as a locus for learning state transitions. Within this region, multivariate patterns of BOLD responses were less correlated between trials with differing input features but similar state associations in the high dimensional than in the low dimensional case, suggesting that these patterns switched between separable (specific to input features) and shared (invariant to input features) transition models. Finally, we show that transition models are updated more strongly following the receipt of positive compared with negative outcomes, a finding that challenges conventional theories of planning. Together, these findings propose a computational and neural account of how information relevant for planning can be shared and segmented in response to the vast array of contextual features we encounter in our world.SIGNIFICANCE STATEMENT Effective planning involves maintaining an accurate model of which actions take us to which locations. But in a world awash with information, mapping actions to states with the right level of complexity is critical. Using a new decision-making "heist task" in conjunction with computational modeling and fMRI, we show that patterns of BOLD responses in the medial temporal lobe-a brain region key for prospective planning-become less sensitive to the presence of visual features when these are irrelevant to the task at hand. By flexibly adapting the complexity of task-state representations in this way, state-action mappings learned under one set of features can be used to plan in the presence of others.
Collapse
Affiliation(s)
- Leonie Glitz
- Department of Experimental Psychology, University of Oxford, Oxford OX2 6HG, United Kingdom
| | - Keno Juechems
- Department of Experimental Psychology, University of Oxford, Oxford OX2 6HG, United Kingdom
| | | | - Neil Garrett
- Department of Experimental Psychology, University of Oxford, Oxford OX2 6HG, United Kingdom
- School of Psychology, University of East Anglia, Norwich NR4 7TJ, United Kingdom
| |
Collapse
|
37
|
Dennison JB, Sazhin D, Smith DV. Decision neuroscience and neuroeconomics: Recent progress and ongoing challenges. WILEY INTERDISCIPLINARY REVIEWS. COGNITIVE SCIENCE 2022; 13:e1589. [PMID: 35137549 PMCID: PMC9124684 DOI: 10.1002/wcs.1589] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/05/2020] [Revised: 11/28/2021] [Accepted: 12/21/2021] [Indexed: 01/10/2023]
Abstract
In the past decade, decision neuroscience and neuroeconomics have developed many new insights in the study of decision making. This review provides an overarching update on how the field has advanced in this time period. Although our initial review a decade ago outlined several theoretical, conceptual, methodological, empirical, and practical challenges, there has only been limited progress in resolving these challenges. We summarize significant trends in decision neuroscience through the lens of the challenges outlined for the field and review examples where the field has had significant, direct, and applicable impacts across economics and psychology. First, we review progress on topics including reward learning, explore-exploit decisions, risk and ambiguity, intertemporal choice, and valuation. Next, we assess the impacts of emotion, social rewards, and social context on decision making. Then, we follow up with how individual differences impact choices and new exciting developments in the prediction and neuroforecasting of future decisions. Finally, we consider how trends in decision-neuroscience research reflect progress toward resolving past challenges, discuss new and exciting applications of recent research, and identify new challenges for the field. This article is categorized under: Psychology > Reasoning and Decision Making Psychology > Emotion and Motivation.
Collapse
Affiliation(s)
- Jeffrey B Dennison
- Department of Psychology, Temple University, Philadelphia, Pennsylvania, USA
| | - Daniel Sazhin
- Department of Psychology, Temple University, Philadelphia, Pennsylvania, USA
| | - David V Smith
- Department of Psychology, Temple University, Philadelphia, Pennsylvania, USA
| |
Collapse
|
38
|
Pallikaras V, Shizgal P. The Convergence Model of Brain Reward Circuitry: Implications for Relief of Treatment-Resistant Depression by Deep-Brain Stimulation of the Medial Forebrain Bundle. Front Behav Neurosci 2022; 16:851067. [PMID: 35431828 PMCID: PMC9011331 DOI: 10.3389/fnbeh.2022.851067] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2022] [Accepted: 02/10/2022] [Indexed: 12/28/2022] Open
Abstract
Deep-brain stimulation of the medial forebrain bundle (MFB) can provide effective, enduring relief of treatment-resistant depression. Panksepp provided an explanatory framework: the MFB constitutes the core of the neural circuitry subserving the anticipation and pursuit of rewards: the “SEEKING” system. On that view, the SEEKING system is hypoactive in depressed individuals; background electrical stimulation of the MFB alleviates symptoms by normalizing activity. Panksepp attributed intracranial self-stimulation to excitation of the SEEKING system in which the ascending projections of midbrain dopamine neurons are an essential component. In parallel with Panksepp’s qualitative work, intracranial self-stimulation has long been studied quantitatively by psychophysical means. That work argues that the predominant directly stimulated substrate for MFB self-stimulation are myelinated, non-dopaminergic fibers, more readily excited by brief electrical current pulses than the thin, unmyelinated axons of the midbrain dopamine neurons. The series-circuit hypothesis reconciles this view with the evidence implicating dopamine in MFB self-stimulation as follows: direct activation of myelinated MFB fibers is rewarding due to their trans-synaptic activation of midbrain dopamine neurons. A recent study in which rats worked for optogenetic stimulation of midbrain dopamine neurons challenges the series-circuit hypothesis and provides a new model of intracranial self-stimulation in which the myelinated non-dopaminergic neurons and the midbrain dopamine projections access the behavioral final common path for reward seeking via separate, converging routes. We explore the potential implications of this convergence model for the interpretation of the antidepressant effect of MFB stimulation. We also discuss the consistent finding that psychomotor stimulants, which boost dopaminergic neurotransmission, fail to provide a monotherapy for depression. We propose that non-dopaminergic MFB components may contribute to the therapeutic effect in parallel to, in synergy with, or even instead of, a dopaminergic component.
Collapse
|
39
|
Song M, Takahashi YK, Burton AC, Roesch MR, Schoenbaum G, Niv Y, Langdon AJ. Minimal cross-trial generalization in learning the representation of an odor-guided choice task. PLoS Comput Biol 2022; 18:e1009897. [PMID: 35333867 PMCID: PMC8986096 DOI: 10.1371/journal.pcbi.1009897] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2021] [Revised: 04/06/2022] [Accepted: 02/04/2022] [Indexed: 11/18/2022] Open
Abstract
There is no single way to represent a task. Indeed, despite experiencing the same task events and contingencies, different subjects may form distinct task representations. As experimenters, we often assume that subjects represent the task as we envision it. However, such a representation cannot be taken for granted, especially in animal experiments where we cannot deliver explicit instruction regarding the structure of the task. Here, we tested how rats represent an odor-guided choice task in which two odor cues indicated which of two responses would lead to reward, whereas a third odor indicated free choice among the two responses. A parsimonious task representation would allow animals to learn from the forced trials what is the better option to choose in the free-choice trials. However, animals may not necessarily generalize across odors in this way. We fit reinforcement-learning models that use different task representations to trial-by-trial choice behavior of individual rats performing this task, and quantified the degree to which each animal used the more parsimonious representation, generalizing across trial types. Model comparison revealed that most rats did not acquire this representation despite extensive experience. Our results demonstrate the importance of formally testing possible task representations that can afford the observed behavior, rather than assuming that animals’ task representations abide by the generative task structure that governs the experimental design. To study how animals learn and make decisions, scientists design experiments, train animals to perform them, and observe how they behave. During this process, an important but rarely asked question is how animals understand the experiment. Merely through observing animals’ behavior in a task, it is often hard to determine if they understand the task in the same way as the experimenter expects. Assuming that animals represent tasks differently than they actually do may lead to incorrect interpretations of behavioral or neural results. Here, we compared different possible representations for a simple reward-learning task in terms of how well these alternative models explain animal’s choice behavior. We found that rats did not represent the task in the most parsimonious way, thereby failing to learn from forced-choice trials what rewards are available on free-choice trials, despite extensive training on the task. These results caution against simply assuming that animals’ understanding of a task corresponds to the way the task was designed.
Collapse
Affiliation(s)
- Mingyu Song
- Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, United States of America
- * E-mail: (MS); (AJL)
| | - Yuji K. Takahashi
- National Institute on Drug Abuse Intramural Research Program, NIH, Baltimore, Maryland, United States of America
| | - Amanda C. Burton
- Department of Psychology, University of Maryland, College Park, Maryland, United States of America
- Program in Neuroscience and Cognitive Science, University of Maryland, College Park, Maryland, United States of America
| | - Matthew R. Roesch
- Department of Psychology, University of Maryland, College Park, Maryland, United States of America
- Program in Neuroscience and Cognitive Science, University of Maryland, College Park, Maryland, United States of America
| | - Geoffrey Schoenbaum
- National Institute on Drug Abuse Intramural Research Program, NIH, Baltimore, Maryland, United States of America
| | - Yael Niv
- Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, United States of America
- Department of Psychology, Princeton University, Princeton, New Jersey, United States of America
| | - Angela J. Langdon
- Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, United States of America
- Department of Psychology, Princeton University, Princeton, New Jersey, United States of America
- * E-mail: (MS); (AJL)
| |
Collapse
|
40
|
Stress-sensitive inference of task controllability. Nat Hum Behav 2022; 6:812-822. [PMID: 35273354 DOI: 10.1038/s41562-022-01306-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Accepted: 01/17/2022] [Indexed: 11/08/2022]
Abstract
Estimating the controllability of the environment enables agents to better predict upcoming events and decide when to engage controlled action selection. How does the human brain estimate controllability? Trial-by-trial analysis of choices, decision times and neural activity in an explore-and-predict task demonstrate that humans solve this problem by comparing the predictions of an 'actor' model with those of a reduced 'spectator' model of their environment. Neural blood oxygen level-dependent responses within striatal and medial prefrontal areas tracked the instantaneous difference in the prediction errors generated by these two statistical learning models. Blood oxygen level-dependent activity in the posterior cingulate, temporoparietal and prefrontal cortices covaried with changes in estimated controllability. Exposure to inescapable stressors biased controllability estimates downward and increased reliance on the spectator model in an anxiety-dependent fashion. Taken together, these findings provide a mechanistic account of controllability inference and its distortion by stress exposure.
Collapse
|
41
|
Grossman CD, Bari BA, Cohen JY. Serotonin neurons modulate learning rate through uncertainty. Curr Biol 2022; 32:586-599.e7. [PMID: 34936883 PMCID: PMC8825708 DOI: 10.1016/j.cub.2021.12.006] [Citation(s) in RCA: 40] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Revised: 10/11/2021] [Accepted: 12/03/2021] [Indexed: 12/20/2022]
Abstract
Regulating how fast to learn is critical for flexible behavior. Learning about the consequences of actions should be slow in stable environments, but accelerate when that environment changes. Recognizing stability and detecting change are difficult in environments with noisy relationships between actions and outcomes. Under these conditions, theories propose that uncertainty can be used to modulate learning rates ("meta-learning"). We show that mice behaving in a dynamic foraging task exhibit choice behavior that varied as a function of two forms of uncertainty estimated from a meta-learning model. The activity of dorsal raphe serotonin neurons tracked both types of uncertainty in the foraging task as well as in a dynamic Pavlovian task. Reversible inhibition of serotonin neurons in the foraging task reproduced changes in learning predicted by a simulated lesion of meta-learning in the model. We thus provide a quantitative link between serotonin neuron activity, learning, and decision making.
Collapse
Affiliation(s)
- Cooper D Grossman
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Bilal A Bari
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA.
| |
Collapse
|
42
|
Karayanni M, Nelken I. Extrinsic rewards, intrinsic rewards, and non-optimal behavior. J Comput Neurosci 2022; 50:139-143. [PMID: 35122189 DOI: 10.1007/s10827-022-00813-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2021] [Revised: 01/20/2022] [Accepted: 01/25/2022] [Indexed: 10/19/2022]
Abstract
The optimality of behavior in experimental settings is usually determined with respect to an extrinsic reward defined by the experimenters. However, actions that do not lead to reward are ubiquitous in many species and in many experimental paradigms. Modern research on decision processes commonly treat non-optimal behaviors as noise, often excluding from analysis animals that do not reach behavioral performance criteria. However, non-optimal behaviors can be a window on important brain processes. Here we explore the evidence that non-optimal behaviors are the consequence of intrinsically motivated actions, related to drives that are different from that of obtaining extrinsic reward. One way of operationally characterizing these drives is by postulating intrinsic rewards associated with them. Behaviors that are apparently non-optimal can be interpreted as the consequence of optimal decisions whose goal is to optimize a combination of intrinsic and extrinsic rewards. We review intrinsic rewards that have been discussed in the literature, and suggest ways of testing their existence and role in shaping animal behavior.
Collapse
Affiliation(s)
- Mousa Karayanni
- Edmond and Lily Safra Center for Brain Sciences, Hebrew University, Jerusalem, Israel.
| | - Israel Nelken
- Edmond and Lily Safra Center for Brain Sciences, Hebrew University, Jerusalem, Israel.,Department of Neurobiology, Silberman Institute of Life Sciences, Hebrew University, Jerusalem, Israel
| |
Collapse
|
43
|
Abstract
Recent breakthroughs in artificial intelligence (AI) have enabled machines to plan in tasks previously thought to be uniquely human. Meanwhile, the planning algorithms implemented by the brain itself remain largely unknown. Here, we review neural and behavioral data in sequential decision-making tasks that elucidate the ways in which the brain does-and does not-plan. To systematically review available biological data, we create a taxonomy of planning algorithms by summarizing the relevant design choices for such algorithms in AI. Across species, recording techniques, and task paradigms, we find converging evidence that the brain represents future states consistent with a class of planning algorithms within our taxonomy-focused, depth-limited, and serial. However, we argue that current data are insufficient for addressing more detailed algorithmic questions. We propose a new approach leveraging AI advances to drive experiments that can adjudicate between competing candidate algorithms.
Collapse
|
44
|
Brunec IK, Momennejad I. Predictive Representations in Hippocampal and Prefrontal Hierarchies. J Neurosci 2022; 42:299-312. [PMID: 34799416 PMCID: PMC8802932 DOI: 10.1523/jneurosci.1327-21.2021] [Citation(s) in RCA: 28] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2021] [Revised: 10/19/2021] [Accepted: 10/22/2021] [Indexed: 11/21/2022] Open
Abstract
As we navigate the world, we use learned representations of relational structures to explore and to reach goals. Studies of how relational knowledge enables inference and planning are typically conducted in controlled small-scale settings. It remains unclear, however, how people use stored knowledge in continuously unfolding navigation (e.g., walking long distances in a city). We hypothesized that multiscale predictive representations guide naturalistic navigation in humans, and these scales are organized along posterior-anterior prefrontal and hippocampal hierarchies. We conducted model-based representational similarity analyses of neuroimaging data collected while male and female participants navigated realistically long paths in virtual reality. We tested the pattern similarity of each point, along each path, to a weighted sum of its successor points within predictive horizons of different scales. We found that anterior PFC showed the largest predictive horizons, posterior hippocampus the smallest, with the anterior hippocampus and orbitofrontal regions in between. Our findings offer novel insights into how cognitive maps support hierarchical planning at multiple scales.SIGNIFICANCE STATEMENT Whenever we navigate the world, we represent our journey at multiple horizons: from our immediate surroundings to our distal goal. How are such cognitive maps at different horizons simultaneously represented in the brain? Here, we applied a reinforcement learning-based analysis to neuroimaging data acquired while participants virtually navigated their hometown. We investigated neural patterns in the hippocampus and PFC, key cognitive map regions. We uncovered predictive representations with multiscale horizons in prefrontal and hippocampal gradients, with the longest predictive horizons in anterior PFC and the shortest in posterior hippocampus. These findings provide empirical support for the computational hypothesis that multiscale neural representations guide goal-directed navigation. This advances our understanding of hierarchical planning in everyday navigation of realistic distances.
Collapse
Affiliation(s)
- Iva K Brunec
- Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104
| | | |
Collapse
|
45
|
Averbeck B, O'Doherty JP. Reinforcement-learning in fronto-striatal circuits. Neuropsychopharmacology 2022; 47:147-162. [PMID: 34354249 PMCID: PMC8616931 DOI: 10.1038/s41386-021-01108-0] [Citation(s) in RCA: 34] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Revised: 07/06/2021] [Accepted: 07/09/2021] [Indexed: 01/03/2023]
Abstract
We review the current state of knowledge on the computational and neural mechanisms of reinforcement-learning with a particular focus on fronto-striatal circuits. We divide the literature in this area into five broad research themes: the target of the learning-whether it be learning about the value of stimuli or about the value of actions; the nature and complexity of the algorithm used to drive the learning and inference process; how learned values get converted into choices and associated actions; the nature of state representations, and of other cognitive machinery that support the implementation of various reinforcement-learning operations. An emerging fifth area focuses on how the brain allocates or arbitrates control over different reinforcement-learning sub-systems or "experts". We will outline what is known about the role of the prefrontal cortex and striatum in implementing each of these functions. We then conclude by arguing that it will be necessary to build bridges from algorithmic level descriptions of computational reinforcement-learning to implementational level models to better understand how reinforcement-learning emerges from multiple distributed neural networks in the brain.
Collapse
Affiliation(s)
| | - John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA.
| |
Collapse
|
46
|
Hocker DL, Brody CD, Savin C, Constantinople CM. Subpopulations of neurons in lOFC encode previous and current rewards at time of choice. eLife 2021; 10:e70129. [PMID: 34693908 PMCID: PMC8616578 DOI: 10.7554/elife.70129] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2021] [Accepted: 10/20/2021] [Indexed: 01/17/2023] Open
Abstract
Studies of neural dynamics in lateral orbitofrontal cortex (lOFC) have shown that subsets of neurons that encode distinct aspects of behavior, such as value, may project to common downstream targets. However, it is unclear whether reward history, which may subserve lOFC's well-documented role in learning, is represented by functional subpopulations in lOFC. Previously, we analyzed neural recordings from rats performing a value-based decision-making task, and we documented trial-by-trial learning that required lOFC (Constantinople et al., 2019). Here, we characterize functional subpopulations of lOFC neurons during behavior, including their encoding of task variables. We found five distinct clusters of lOFC neurons, either based on clustering of their trial-averaged peristimulus time histograms (PSTHs), or a feature space defined by their average conditional firing rates aligned to different task variables. We observed weak encoding of reward attributes, but stronger encoding of reward history, the animal's left or right choice, and reward receipt across all clusters. Only one cluster, however, encoded the animal's reward history at the time shortly preceding the choice, suggesting a possible role in integrating previous and current trial outcomes at the time of choice. This cluster also exhibits qualitatively similar responses to identified corticostriatal projection neurons in a recent study (Hirokawa et al., 2019), and suggests a possible role for subpopulations of lOFC neurons in mediating trial-by-trial learning.
Collapse
Affiliation(s)
- David L Hocker
- Center for Neural Science, New York UniversityNew YorkUnited States
| | - Carlos D Brody
- Center for Neural Science, New York UniversityNew YorkUnited States
- Princeton Neuroscience Institute, Princeton UniversityPrincetonUnited States
- Department of Molecular Biology, Princeton UniversityPrincetonUnited States
- Howard Hughes Medical Institute, Princeton UniversityPrincetonUnited States
| | - Cristina Savin
- Center for Neural Science, New York UniversityNew YorkUnited States
- Center for Data Science, New York UniversityNew YorkUnited States
| | | |
Collapse
|
47
|
Abstract
In novel situations, where direct experience is lacking or outdated, humans must rely on mental simulations to predict future outcomes. This review discusses recent work on the neural circuits that support such inference-based behavior. We focus on two specific examples: 1) using knowledge about the associative structure of the world to infer outcomes when direct experience is lacking; 2) inferring the current value of options when the desirability of the associated outcome has changed since the original learning experience. These two examples can be studied in the sensory preconditioning and devaluation tasks, respectively. We review results from studies in animals and humans suggesting that the orbitofrontal cortex (OFC), together with the hippocampus and amygdala, is necessary for inference in both of these tasks. Together, these findings suggest that the OFC is a critical hub in the brain network that supports inference-based decision-making.
Collapse
Affiliation(s)
- Fang Wang
- Department of Neurology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA
- Center for Cognitive Neuroscience, Duke University, Durham, North Carolina, USA
| | - Thorsten Kahnt
- Department of Neurology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA
- Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, USA
- Department of Psychology, Weinberg College of Arts and Sciences, Northwestern University, Evanston, Illinois, USA
| |
Collapse
|
48
|
Banker SM, Gu X, Schiller D, Foss-Feig JH. Hippocampal contributions to social and cognitive deficits in autism spectrum disorder. Trends Neurosci 2021; 44:793-807. [PMID: 34521563 DOI: 10.1016/j.tins.2021.08.005] [Citation(s) in RCA: 99] [Impact Index Per Article: 33.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2021] [Revised: 07/07/2021] [Accepted: 08/10/2021] [Indexed: 10/20/2022]
Abstract
Autism spectrum disorder (ASD) is characterized by hallmark impairments in social functioning. Nevertheless, nonsocial cognition, including hippocampus-dependent spatial reasoning and episodic memory, is also commonly impaired in ASD. ASD symptoms typically emerge between 12 and 24 months of age, a time window associated with critical developmental events in the hippocampus. Despite this temporal overlap and evidence of hippocampal structural abnormalities in ASD individuals, relatively few human studies have focused on hippocampal function in ASD. Herein, we review the existing evidence for the involvement of the hippocampus in ASD and highlight the hippocampus as a promising area of interest for future research in ASD.
Collapse
Affiliation(s)
- Sarah M Banker
- Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; Seaver Autism Center for Research and Treatment, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; Center for Computational Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; The Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA.
| | - Xiaosi Gu
- Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; Center for Computational Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; The Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA
| | - Daniela Schiller
- Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; Center for Computational Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; The Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA
| | - Jennifer H Foss-Feig
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; Seaver Autism Center for Research and Treatment, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; The Mindich Child Health and Development Institute, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA; The Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA
| |
Collapse
|
49
|
Dombrovski AY, Hallquist MN. Search for solutions, learning, simulation, and choice processes in suicidal behavior. WILEY INTERDISCIPLINARY REVIEWS. COGNITIVE SCIENCE 2021; 13:e1561. [PMID: 34008338 PMCID: PMC9285563 DOI: 10.1002/wcs.1561] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/12/2020] [Revised: 03/06/2021] [Accepted: 04/07/2021] [Indexed: 12/25/2022]
Abstract
Suicide may be viewed as an unfortunate outcome of failures in decision processes. Such failures occur when the demands of a crisis exceed a person's capacity to (i) search for options, (ii) learn and simulate possible futures, and (iii) make advantageous value‐based choices. Can individual‐level decision deficits and biases drive the progression of the suicidal crisis? Our overview of the evidence on this question is informed by clinical theory and grounded in reinforcement learning and behavioral economics. Cohort and case–control studies provide strong evidence that limited cognitive capacity and particularly impaired cognitive control are associated with suicidal behavior, imposing cognitive constraints on decision‐making. We conceptualize suicidal ideation as an element of impoverished consideration sets resulting from a search for solutions under cognitive constraints and mood‐congruent Pavlovian influences, a view supported by mostly indirect evidence. More compelling is the evidence of impaired learning in people with a history of suicidal behavior. We speculate that an inability to simulate alternative futures using one's model of the world may undermine alternative solutions in a suicidal crisis. The hypothesis supported by the strongest evidence is that the selection of suicide over alternatives is facilitated by a choice process undermined by randomness. Case–control studies using gambling tasks, armed bandits, and delay discounting support this claim. Future experimental studies will need to uncover real‐time dynamics of choice processes in suicidal people. In summary, the decision process framework sheds light on neurocognitive mechanisms that facilitate the progression of the suicidal crisis. This article is categorized under:Economics > Individual Decision‐Making Psychology > Emotion and Motivation Psychology > Learning Neuroscience > Behavior
Collapse
Affiliation(s)
| | - Michael N Hallquist
- Department of Psychology and Neuroscience, University of North Carolina, Chapel Hill, North Carolina, USA
| |
Collapse
|
50
|
Heo S, Sung Y, Lee SW. Effects of subclinical depression on prefrontal-striatal model-based and model-free learning. PLoS Comput Biol 2021; 17:e1009003. [PMID: 33989284 PMCID: PMC8153417 DOI: 10.1371/journal.pcbi.1009003] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2020] [Revised: 05/26/2021] [Accepted: 04/26/2021] [Indexed: 11/18/2022] Open
Abstract
Depression is characterized by deficits in the reinforcement learning (RL) process. Although many computational and neural studies have extended our knowledge of the impact of depression on RL, most focus on habitual control (model-free RL), yielding a relatively poor understanding of goal-directed control (model-based RL) and arbitration control to find a balance between the two. We investigated the effects of subclinical depression on model-based and model-free learning in the prefrontal-striatal circuitry. First, we found that subclinical depression is associated with the attenuated state and reward prediction error representation in the insula and caudate. Critically, we found that it accompanies the disrupted arbitration control between model-based and model-free learning in the predominantly inferior lateral prefrontal cortex and frontopolar cortex. We also found that depression undermines the ability to exploit viable options, called exploitation sensitivity. These findings characterize how subclinical depression influences different levels of the decision-making hierarchy, advancing previous conflicting views that depression simply influences either habitual or goal-directed control. Our study creates possibilities for various clinical applications, such as early diagnosis and behavioral therapy design.
Collapse
Affiliation(s)
- Suyeon Heo
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- Brain and Cognitive Engineering Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Yoondo Sung
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- Brain and Cognitive Engineering Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Institute for Health Science Technology, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Institute for Artificial Intelligence, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Center for Neuroscience-inspired AI, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| |
Collapse
|